2024-04-15T15:09:31.669 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T15:09:31.670 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144 -- /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/orig.config.yaml 2024-04-15T15:09:31.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T15:09:31.798 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144 branch: wip-lusov-quiesce-xlock description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} email: null first_in_suite: false job_id: '7657144' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiesce-xlock ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow request sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiesce-xlock sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5929 sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 sleep_before_teardown: 0 suite: fs suite_branch: wip-lusov-quiesce-xlock suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIE9jX1BIR5LrfW9u7p9ZFWARk8rZM4rxQ7Pj9xRDsPwJAkFNbl74+1mqPiy2sFSYQdawFpR9IBX0HvGSLEKDHk= smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxUayo0doJBrH/sTt93Zv9BrOxzjdCJzWs0DlBISUHJ16DVvUvYhyj5tpb4iCkQc063omFSnO4e+x1yQTn2nKg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_quiesce teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_14:54:08 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-15T15:09:31.798 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa; will attempt to use it 2024-04-15T15:09:31.799 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks 2024-04-15T15:09:31.799 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-15T15:09:31.801 INFO:teuthology.task.internal:Checking packages... 2024-04-15T15:09:31.834 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14' 2024-04-15T15:09:31.835 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T15:09:31.835 INFO:teuthology.packaging:ref: None 2024-04-15T15:09:31.835 INFO:teuthology.packaging:tag: None 2024-04-15T15:09:31.835 INFO:teuthology.packaging:branch: wip-lusov-quiesce-xlock 2024-04-15T15:09:31.835 INFO:teuthology.packaging:sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:09:31.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiesce-xlock 2024-04-15T15:09:31.983 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2861.g9bf1c4ba 2024-04-15T15:09:31.984 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-15T15:09:31.991 INFO:teuthology.task.internal:no buildpackages task found 2024-04-15T15:09:31.992 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T15:09:32.031 INFO:teuthology.task.internal:Saving configuration 2024-04-15T15:09:32.050 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T15:09:32.055 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:09:32.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 15:04:59.841715', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIE9jX1BIR5LrfW9u7p9ZFWARk8rZM4rxQ7Pj9xRDsPwJAkFNbl74+1mqPiy2sFSYQdawFpR9IBX0HvGSLEKDHk='} 2024-04-15T15:09:32.104 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 15:04:59.843113', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxUayo0doJBrH/sTt93Zv9BrOxzjdCJzWs0DlBISUHJ16DVvUvYhyj5tpb4iCkQc063omFSnO4e+x1yQTn2nKg='} 2024-04-15T15:09:32.105 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T15:09:32.112 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-15T15:09:32.112 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-15T15:09:32.112 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T15:09:32.205 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f95fc48f0d0>, signals=[15]) 2024-04-15T15:09:32.206 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T15:09:32.213 INFO:teuthology.task.internal:Opening connections... 2024-04-15T15:09:32.214 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:09:32.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:09:32.322 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:09:32.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:09:32.428 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T15:09:32.435 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-04-15T15:09:32.453 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-04-15T15:09:32.453 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-04-15T15:09:32.514 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-04-15T15:09:32.515 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T15:09:32.515 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-04-15T15:09:32.515 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T15:09:32.515 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T15:09:32.515 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T15:09:32.515 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-04-15T15:09:32.540 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2024-04-15T15:09:32.559 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2024-04-15T15:09:32.559 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2024-04-15T15:09:32.617 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2024-04-15T15:09:32.617 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T15:09:32.618 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2024-04-15T15:09:32.619 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T15:09:32.619 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T15:09:32.619 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T15:09:32.619 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-04-15T15:09:32.639 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T15:09:32.649 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T15:09:32.672 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T15:09:32.673 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-04-15T15:09:32.675 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2024-04-15T15:09:32.692 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T15:09:32.699 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T15:09:32.700 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-04-15T15:09:32.737 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2024-04-15T15:09:32.763 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T15:09:32.819 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T15:09:32.837 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-15T15:09:32.837 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-15T15:09:32.837 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T15:09:32.838 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T15:09:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:09:32.854 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-04-15T15:09:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:09:32.856 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-04-15T15:09:32.914 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-361.el9.x86_64 2024-04-15T15:09:32.915 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.14.0-361.el9.x86_64 2024-04-15T15:09:32.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2024-04-15T15:09:32.917 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-361.el9.x86_64 2024-04-15T15:09:32.917 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.14.0-361.el9.x86_64 2024-04-15T15:09:32.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2024-04-15T15:09:35.427 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-04-15T15:09:35.832 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 5.4 MB/s | 6.7 MB 00:01 2024-04-15T15:09:36.877 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-04-15T15:09:37.431 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-04-15T15:09:42.788 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:05 2024-04-15T15:09:46.700 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 886 kB/s | 8.0 MB 00:09 2024-04-15T15:09:49.117 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-15T15:09:53.234 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-15T15:09:54.760 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-04-15T15:09:55.903 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-04-15T15:10:01.252 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-15T15:10:03.717 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T15:10:03.718 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T15:10:03.808 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-04-15T15:10:03.818 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-15T15:10:03.818 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-04-15T15:10:03.819 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-15T15:10:03.819 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-04-15T15:10:03.819 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T15:10:03.819 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-04-15T15:10:03.819 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T15:10:03.819 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T15:10:03.820 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T15:10:03.820 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:10:03.820 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-04-15T15:10:03.820 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-15T15:10:03.820 INFO:teuthology.orchestra.run.smithi012.stdout:Install 4 Packages 2024-04-15T15:10:03.820 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:10:03.821 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 77 M 2024-04-15T15:10:03.821 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 123 M 2024-04-15T15:10:03.821 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-04-15T15:10:04.128 INFO:teuthology.orchestra.run.smithi012.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 897 kB/s | 277 kB 00:00 2024-04-15T15:10:05.229 INFO:teuthology.orchestra.run.smithi012.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 11 MB/s | 15 MB 00:01 2024-04-15T15:10:05.829 INFO:teuthology.orchestra.run.smithi012.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 16 MB/s | 28 MB 00:01 2024-04-15T15:10:06.746 INFO:teuthology.orchestra.run.smithi012.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 12 MB/s | 34 MB 00:02 2024-04-15T15:10:06.749 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:10:06.749 INFO:teuthology.orchestra.run.smithi012.stdout:Total 26 MB/s | 77 MB 00:02 2024-04-15T15:10:06.749 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-04-15T15:10:07.197 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-04-15T15:10:07.197 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-04-15T15:10:08.065 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-04-15T15:10:08.065 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-04-15T15:10:08.912 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 774 kB/s | 19 MB 00:24 2024-04-15T15:10:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-04-15T15:10:11.167 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:10:11.254 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:10:14.135 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:10:14.266 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:10:14.687 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.2 MB 00:02 2024-04-15T15:10:16.429 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-15T15:10:19.331 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:10:19.373 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:10:24.477 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:10:43.015 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 806 kB/s | 21 MB 00:26 2024-04-15T15:10:48.154 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-04-15T15:10:50.627 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T15:10:50.627 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T15:10:50.724 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-04-15T15:10:50.733 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-04-15T15:10:50.734 INFO:teuthology.orchestra.run.smithi055.stdout:Install 4 Packages 2024-04-15T15:10:50.734 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:10:50.734 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 77 M 2024-04-15T15:10:50.734 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 123 M 2024-04-15T15:10:50.734 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-04-15T15:10:50.888 INFO:teuthology.orchestra.run.smithi055.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.8 MB/s | 277 kB 00:00 2024-04-15T15:10:51.530 INFO:teuthology.orchestra.run.smithi055.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-04-15T15:10:52.080 INFO:teuthology.orchestra.run.smithi055.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-04-15T15:10:52.339 INFO:teuthology.orchestra.run.smithi055.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-15T15:10:52.343 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:10:52.343 INFO:teuthology.orchestra.run.smithi055.stdout:Total 48 MB/s | 77 MB 00:01 2024-04-15T15:10:52.343 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-04-15T15:10:52.787 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-04-15T15:10:52.788 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-04-15T15:10:53.652 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-04-15T15:10:53.653 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-04-15T15:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T15:10:55.224 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:10:55.270 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:10:55.672 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:10:55.672 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:10:55.672 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:10:55.672 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:10:56.067 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-04-15T15:10:56.091 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-04-15T15:10:56.385 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:10:56.484 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:10:56.484 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:10:56.484 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T15:10:56.484 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T15:10:56.485 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T15:10:56.485 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-15T15:10:56.485 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T15:10:56.485 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:10:56.568 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:10:56.568 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:10:56.568 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-04-15T15:10:56.606 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-04-15T15:10:56.606 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-04-15T15:10:56.606 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2024-04-15T15:10:56.606 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-04-15T15:10:56.607 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:10:56.608 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T15:10:56.609 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T15:10:56.609 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 5.14.0-438.el9.x86_64 2024-04-15T15:10:56.609 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T15:10:56.684 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:10:56.765 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:10:57.199 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-04-15T15:10:58.570 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T15:10:58.597 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-04-15T15:10:58.600 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-04-15T15:10:58.615 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.abtspCIxlc 2024-04-15T15:10:58.615 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.abtspCIxlc 2024-04-15T15:10:58.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.abtspCIxlc 2024-04-15T15:10:58.837 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.abtspCIxlc is 6KB 2024-04-15T15:10:58.886 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.abtspCIxlc 2024-04-15T15:10:58.901 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T15:10:58.979 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T15:10:58.979 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T15:10:58.979 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T15:10:58.979 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T15:10:58.980 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T15:10:59.119 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-04-15T15:10:59.649 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:10:59.785 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:11:04.858 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:11:04.909 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:11:10.003 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:11:29.122 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-15T15:11:29.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-15T15:11:29.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:11:41.008 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:11:41.008 INFO:teuthology.orchestra.run.smithi055.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T15:11:41.008 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:11:41.051 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:11:41.454 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:11:41.454 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:11:41.454 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:11:41.454 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:11:41.883 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-04-15T15:11:42.214 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:11:42.319 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:11:42.320 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:11:42.320 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T15:11:42.320 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T15:11:42.320 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T15:11:42.320 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-15T15:11:42.320 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T15:11:42.320 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:11:42.403 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:11:42.404 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:11:42.404 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2024-04-15T15:11:42.441 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2024-04-15T15:11:42.441 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2024-04-15T15:11:42.441 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.06 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 68.el9 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Group : Unspecified 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 8264254 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : builder@centos.org 2024-04-15T15:11:42.442 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:11:42.443 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T15:11:42.444 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T15:11:42.444 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 5.14.0-438.el9.x86_64 2024-04-15T15:11:42.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T15:11:43.093 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2024-04-15T15:11:44.459 INFO:teuthology.orchestra.run.smithi055.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T15:11:44.500 INFO:teuthology.orchestra.run.smithi055.stderr:done 2024-04-15T15:11:44.505 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-04-15T15:11:44.521 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.LJA4oAJg9v 2024-04-15T15:11:44.521 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LJA4oAJg9v 2024-04-15T15:11:44.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.LJA4oAJg9v 2024-04-15T15:11:44.737 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.LJA4oAJg9v is 6KB 2024-04-15T15:11:44.787 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.LJA4oAJg9v 2024-04-15T15:11:44.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T15:11:44.870 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T15:11:44.871 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T15:11:44.871 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T15:11:44.871 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T15:11:44.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T15:11:45.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2024-04-15T15:11:47.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-04-15T15:11:56.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-15T15:11:56.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:12:15.038 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-15T15:12:15.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-04-15T15:12:15.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:12:33.571 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2024-04-15T15:12:42.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-04-15T15:12:42.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:12:56.511 DEBUG:teuthology.orchestra.remote:timed out 2024-04-15T15:13:08.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-15T15:13:08.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:13:08.840 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-04-15T15:13:09.245 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-15T15:13:09.245 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-15T15:13:09.245 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-04-15T15:13:09.261 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-438.el9.x86_64 2024-04-15T15:13:09.261 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T15:13:09.262 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T15:13:09.262 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T15:13:10.262 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-15T15:13:10.263 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T15:13:10.338 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-04-15T15:13:10.370 DEBUG:teuthology.parallel:result is None 2024-04-15T15:13:14.346 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-04-15T15:13:14.854 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-04-15T15:13:14.855 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-438.el9.x86_64"... 2024-04-15T15:13:14.855 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-04-15T15:13:14.869 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-438.el9.x86_64 2024-04-15T15:13:14.869 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T15:13:14.869 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T15:13:14.869 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T15:13:15.871 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-15T15:13:15.871 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T15:13:15.984 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2024-04-15T15:13:16.032 DEBUG:teuthology.parallel:result is None 2024-04-15T15:13:16.032 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T15:13:16.040 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T15:13:16.041 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T15:13:16.043 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T15:13:16.060 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T15:13:16.088 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T15:13:16.121 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T15:13:16.121 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T15:13:16.124 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T15:13:16.155 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T15:13:16.163 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T15:13:16.163 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T15:13:16.191 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T15:13:16.228 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:13:16.243 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:13:16.243 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:13:16.257 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:13:16.259 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T15:13:16.266 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T15:13:16.267 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T15:13:16.288 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T15:13:16.317 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T15:13:16.341 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T15:13:16.341 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T15:13:16.360 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T15:13:16.375 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:13:16.465 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:13:16.513 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:13:16.596 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:13:16.644 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:13:16.644 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T15:13:16.714 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:13:16.757 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:13:16.805 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:13:16.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:13:16.933 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:13:16.933 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T15:13:17.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-04-15T15:13:17.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2024-04-15T15:13:17.051 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:13:17.088 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:13:17.512 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T15:13:17.524 INFO:teuthology.task.internal:Starting timer... 2024-04-15T15:13:17.524 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T15:13:17.576 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T15:13:17.604 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-15T15:13:17.604 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-04-15T15:13:17.658 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-04-15T15:13:17.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2024-04-15T15:13:17.716 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2024-04-15T15:13:17.717 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T15:13:17.718 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-04-15T15:13:17.747 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-04-15T15:13:17.747 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2024-04-15T15:13:17.769 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2024-04-15T15:13:17.769 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi055.front.sepia.ceph.com': 'permissive'} 2024-04-15T15:13:17.770 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:13:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:13:17.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:13:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:13:17.849 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T15:13:17.849 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-04-15T15:13:17.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2024-04-15T15:13:17.937 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T15:13:17.960 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T15:13:17.977 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T15:13:17.982 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi055.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T15:17:41.854 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi055.front.sepia.ceph.com')] 2024-04-15T15:17:41.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-15T15:17:41.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:17:41.995 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-04-15T15:17:42.074 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-15T15:17:42.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-04-15T15:17:42.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:17:42.172 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-04-15T15:17:42.250 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-04-15T15:17:42.251 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T15:17:42.262 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T15:17:42.262 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T15:17:42.263 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:17:42.265 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T15:17:42.265 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:17:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T15:17:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T15:17:42.324 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T15:17:42.342 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T15:17:42.348 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-04-15T15:17:42.373 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2024-04-15T15:17:42.380 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T15:17:42.388 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2024-04-15T15:17:42.397 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T15:17:42.405 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T15:17:42.422 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T15:17:42.450 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:17:42.453 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:17:42.453 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-04-15T15:17:42.453 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:42.454 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:42.454 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:42.454 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:42.475 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:17:42.478 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:17:42.478 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-04-15T15:17:42.478 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:42.478 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:42.479 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:42.479 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:42.480 INFO:teuthology.run_tasks:Running task install... 2024-04-15T15:17:42.489 DEBUG:teuthology.task.install:project ceph 2024-04-15T15:17:42.489 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}} 2024-04-15T15:17:42.489 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'} 2024-04-15T15:17:42.489 INFO:teuthology.task.install:Using flavor: default 2024-04-15T15:17:42.495 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-15T15:17:42.496 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-15T15:17:42.496 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'tag': None, 'wait_for_package': False} 2024-04-15T15:17:42.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:17:42.498 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'tag': None, 'wait_for_package': False} 2024-04-15T15:17:42.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:17:42.646 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/ 2024-04-15T15:17:42.646 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2861.g9bf1c4ba 2024-04-15T15:17:42.735 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/ 2024-04-15T15:17:42.735 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2861.g9bf1c4ba 2024-04-15T15:17:42.876 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T15:17:42.876 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:17:42.877 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T15:17:42.907 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-15T15:17:42.907 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T15:17:42.959 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T15:17:42.959 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:17:42.959 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T15:17:42.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T15:17:42.992 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-15T15:17:42.992 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T15:17:43.061 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T15:17:43.063 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T15:17:43.129 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2024-04-15T15:17:43.130 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2024-04-15T15:17:43.147 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T15:17:43.175 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-04-15T15:17:43.177 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-04-15T15:17:43.387 INFO:teuthology.orchestra.run.smithi055.stdout:74 files removed 2024-04-15T15:17:43.415 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-15T15:17:43.420 INFO:teuthology.orchestra.run.smithi012.stdout:74 files removed 2024-04-15T15:17:43.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-15T15:17:44.086 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 254 kB/s | 83 kB 00:00 2024-04-15T15:17:44.161 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 191 kB/s | 83 kB 00:00 2024-04-15T15:17:44.728 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 19 kB/s | 11 kB 00:00 2024-04-15T15:17:44.778 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 20 kB/s | 11 kB 00:00 2024-04-15T15:17:45.295 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-04-15T15:17:46.214 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-04-15T15:17:47.065 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 3.9 MB/s | 6.7 MB 00:01 2024-04-15T15:17:47.707 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-04-15T15:17:48.514 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-04-15T15:17:49.748 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-04-15T15:17:49.755 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 8.0 MB 00:00 2024-04-15T15:17:52.632 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-15T15:17:54.714 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-04-15T15:17:58.881 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-15T15:18:00.582 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-15T15:18:05.199 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 715 kB/s | 6.2 MB 00:08 2024-04-15T15:18:07.000 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-15T15:18:20.425 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-04-15T15:18:25.655 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-04-15T15:18:25.678 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-04-15T15:18:28.453 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:18:28.453 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:18:28.469 INFO:teuthology.orchestra.run.smithi012.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-15T15:18:28.470 INFO:teuthology.orchestra.run.smithi012.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-15T15:18:28.472 INFO:teuthology.orchestra.run.smithi012.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-15T15:18:28.473 INFO:teuthology.orchestra.run.smithi012.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-15T15:18:28.475 INFO:teuthology.orchestra.run.smithi012.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-15T15:18:28.478 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-15T15:18:28.479 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-15T15:18:28.566 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-04-15T15:18:28.583 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-04-15T15:18:28.583 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-04-15T15:18:28.583 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-04-15T15:18:28.583 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-04-15T15:18:28.583 INFO:teuthology.orchestra.run.smithi012.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-15T15:18:28.583 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 6.6 k 2024-04-15T15:18:28.583 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.4 M 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 890 k 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 145 k 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 1.5 M 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 142 k 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 3.5 M 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 7.4 M 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 49 k 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 10 M 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 49 M 2024-04-15T15:18:28.584 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 264 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 762 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 224 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 27 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 18 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 32 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 747 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 127 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-15T15:18:28.585 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 162 k 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 320 k 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 297 k 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 99 k 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 85 k 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 171 k 2024-04-15T15:18:28.586 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-15T15:18:28.587 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.4 M 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-15T15:18:28.588 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 21 M 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 34 k 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 2.4 M 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 249 k 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 4.7 M 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 17 M 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 16 k 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 25 k 2024-04-15T15:18:28.589 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 164 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T15:18:28.590 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 501 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.2 M 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-15T15:18:28.591 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 45 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 135 k 2024-04-15T15:18:28.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-15T15:18:28.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T15:18:28.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T15:18:28.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-15T15:18:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T15:18:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T15:18:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T15:18:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T15:18:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T15:18:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T15:18:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T15:18:28.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T15:18:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T15:18:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T15:18:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T15:18:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-15T15:18:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T15:18:28.597 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T15:18:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T15:18:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T15:18:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-15T15:18:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-15T15:18:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T15:18:28.598 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T15:18:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T15:18:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-15T15:18:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T15:18:28.599 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-15T15:18:28.599 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2024-04-15T15:18:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-15T15:18:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:18:28.600 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-04-15T15:18:28.600 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-04-15T15:18:28.600 INFO:teuthology.orchestra.run.smithi012.stdout:Install 150 Packages 2024-04-15T15:18:28.600 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 24 Packages 2024-04-15T15:18:28.600 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:18:28.601 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 228 M 2024-04-15T15:18:28.602 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-04-15T15:18:29.281 INFO:teuthology.orchestra.run.smithi012.stdout:(1/174): ceph-19.0.0-2861.g9bf1c4ba.el8.x86_64. 30 kB/s | 6.6 kB 00:00 2024-04-15T15:18:29.598 INFO:teuthology.orchestra.run.smithi012.stdout:(2/174): ceph-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 2.7 MB/s | 890 kB 00:00 2024-04-15T15:18:29.690 INFO:teuthology.orchestra.run.smithi012.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 1.5 MB/s | 145 kB 00:00 2024-04-15T15:18:29.848 INFO:teuthology.orchestra.run.smithi012.stdout:(4/174): ceph-base-19.0.0-2861.g9bf1c4ba.el8.x8 6.8 MB/s | 5.4 MB 00:00 2024-04-15T15:18:30.040 INFO:teuthology.orchestra.run.smithi012.stdout:(5/174): ceph-mds-19.0.0-2861.g9bf1c4ba.el8.x86 6.7 MB/s | 2.4 MB 00:00 2024-04-15T15:18:30.082 INFO:teuthology.orchestra.run.smithi012.stdout:(6/174): ceph-mgr-19.0.0-2861.g9bf1c4ba.el8.x86 6.4 MB/s | 1.5 MB 00:00 2024-04-15T15:18:30.708 INFO:teuthology.orchestra.run.smithi012.stdout:(7/174): ceph-mon-19.0.0-2861.g9bf1c4ba.el8.x86 7.0 MB/s | 4.7 MB 00:00 2024-04-15T15:18:30.934 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-15T15:18:31.607 INFO:teuthology.orchestra.run.smithi012.stdout:(8/174): ceph-common-19.0.0-2861.g9bf1c4ba.el8. 8.2 MB/s | 21 MB 00:02 2024-04-15T15:18:31.783 INFO:teuthology.orchestra.run.smithi012.stdout:(9/174): ceph-osd-19.0.0-2861.g9bf1c4ba.el8.x86 9.9 MB/s | 17 MB 00:01 2024-04-15T15:18:31.808 INFO:teuthology.orchestra.run.smithi012.stdout:(10/174): ceph-selinux-19.0.0-2861.g9bf1c4ba.el 125 kB/s | 25 kB 00:00 2024-04-15T15:18:31.967 INFO:teuthology.orchestra.run.smithi012.stdout:(11/174): ceph-radosgw-19.0.0-2861.g9bf1c4ba.el 7.9 MB/s | 10 MB 00:01 2024-04-15T15:18:31.992 INFO:teuthology.orchestra.run.smithi012.stdout:(12/174): cephfs-mirror-19.0.0-2861.g9bf1c4ba.e 1.2 MB/s | 224 kB 00:00 2024-04-15T15:18:32.042 INFO:teuthology.orchestra.run.smithi012.stdout:(13/174): cephfs-shell-19.0.0-2861.g9bf1c4ba.el 360 kB/s | 27 kB 00:00 2024-04-15T15:18:32.084 INFO:teuthology.orchestra.run.smithi012.stdout:(14/174): libcephfs-devel-19.0.0-2861.g9bf1c4ba 344 kB/s | 32 kB 00:00 2024-04-15T15:18:32.168 INFO:teuthology.orchestra.run.smithi012.stdout:(15/174): libcephsqlite-19.0.0-2861.g9bf1c4ba.e 1.9 MB/s | 164 kB 00:00 2024-04-15T15:18:32.193 INFO:teuthology.orchestra.run.smithi012.stdout:(16/174): libcephfs2-19.0.0-2861.g9bf1c4ba.el8. 4.9 MB/s | 747 kB 00:00 2024-04-15T15:18:32.243 INFO:teuthology.orchestra.run.smithi012.stdout:(17/174): librados-devel-19.0.0-2861.g9bf1c4ba. 1.7 MB/s | 127 kB 00:00 2024-04-15T15:18:32.277 INFO:teuthology.orchestra.run.smithi012.stdout:(18/174): libradosstriper1-19.0.0-2861.g9bf1c4b 5.9 MB/s | 501 kB 00:00 2024-04-15T15:18:32.352 INFO:teuthology.orchestra.run.smithi012.stdout:(19/174): python3-ceph-argparse-19.0.0-2861.g9b 602 kB/s | 45 kB 00:00 2024-04-15T15:18:32.444 INFO:teuthology.orchestra.run.smithi012.stdout:(20/174): python3-ceph-common-19.0.0-2861.g9bf1 1.4 MB/s | 135 kB 00:00 2024-04-15T15:18:32.536 INFO:teuthology.orchestra.run.smithi012.stdout:(21/174): python3-cephfs-19.0.0-2861.g9bf1c4ba. 1.7 MB/s | 162 kB 00:00 2024-04-15T15:18:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:(22/174): python3-rados-19.0.0-2861.g9bf1c4ba.e 2.9 MB/s | 320 kB 00:00 2024-04-15T15:18:32.745 INFO:teuthology.orchestra.run.smithi012.stdout:(23/174): librgw2-19.0.0-2861.g9bf1c4ba.el8.x86 10 MB/s | 5.2 MB 00:00 2024-04-15T15:18:32.770 INFO:teuthology.orchestra.run.smithi012.stdout:(24/174): python3-rbd-19.0.0-2861.g9bf1c4ba.el8 2.3 MB/s | 297 kB 00:00 2024-04-15T15:18:32.812 INFO:teuthology.orchestra.run.smithi012.stdout:(25/174): python3-rgw-19.0.0-2861.g9bf1c4ba.el8 1.4 MB/s | 99 kB 00:00 2024-04-15T15:18:32.837 INFO:teuthology.orchestra.run.smithi012.stdout:(26/174): rbd-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 1.2 MB/s | 85 kB 00:00 2024-04-15T15:18:32.921 INFO:teuthology.orchestra.run.smithi012.stdout:(27/174): rbd-nbd-19.0.0-2861.g9bf1c4ba.el8.x86 2.0 MB/s | 171 kB 00:00 2024-04-15T15:18:33.014 INFO:teuthology.orchestra.run.smithi012.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2861.g 372 kB/s | 34 kB 00:00 2024-04-15T15:18:33.130 INFO:teuthology.orchestra.run.smithi012.stdout:(29/174): rbd-mirror-19.0.0-2861.g9bf1c4ba.el8. 9.7 MB/s | 3.1 MB 00:00 2024-04-15T15:18:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2861.g9bf1c4b 898 kB/s | 142 kB 00:00 2024-04-15T15:18:33.556 INFO:teuthology.orchestra.run.smithi012.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2861.g9bf1c 8.3 MB/s | 3.5 MB 00:00 2024-04-15T15:18:33.647 INFO:teuthology.orchestra.run.smithi012.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2861.g9b 2.7 MB/s | 249 kB 00:00 2024-04-15T15:18:33.723 INFO:teuthology.orchestra.run.smithi012.stdout:(33/174): ceph-mgr-rook-19.0.0-2861.g9bf1c4ba.e 657 kB/s | 49 kB 00:00 2024-04-15T15:18:33.738 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:18:33.739 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:18:33.752 INFO:teuthology.orchestra.run.smithi055.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-15T15:18:33.752 INFO:teuthology.orchestra.run.smithi055.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-15T15:18:33.753 INFO:teuthology.orchestra.run.smithi055.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-15T15:18:33.754 INFO:teuthology.orchestra.run.smithi055.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-15T15:18:33.756 INFO:teuthology.orchestra.run.smithi055.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-15T15:18:33.758 INFO:teuthology.orchestra.run.smithi055.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-15T15:18:33.759 INFO:teuthology.orchestra.run.smithi055.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-15T15:18:33.798 INFO:teuthology.orchestra.run.smithi012.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2861.g9 220 kB/s | 16 kB 00:00 2024-04-15T15:18:33.844 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-04-15T15:18:33.861 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 6.6 k 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.4 M 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 890 k 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 145 k 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 1.5 M 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 142 k 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 3.5 M 2024-04-15T15:18:33.862 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 7.4 M 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 49 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 10 M 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 49 M 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 264 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 762 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 224 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-shell x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 27 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-top noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 18 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-15T15:18:33.863 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 32 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 747 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 127 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 162 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 320 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 297 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 99 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 85 k 2024-04-15T15:18:33.864 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 171 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-15T15:18:33.865 INFO:teuthology.orchestra.run.smithi055.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.4 M 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-15T15:18:33.866 INFO:teuthology.orchestra.run.smithi055.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 21 M 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 34 k 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 2.4 M 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 249 k 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 4.7 M 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 17 M 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 16 k 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 25 k 2024-04-15T15:18:33.867 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 164 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T15:18:33.868 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 501 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.2 M 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-15T15:18:33.869 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 45 k 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 135 k 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T15:18:33.870 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T15:18:33.871 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T15:18:33.872 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T15:18:33.873 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T15:18:33.874 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-04-15T15:18:33.875 INFO:teuthology.orchestra.run.smithi055.stdout:Install 150 Packages 2024-04-15T15:18:33.876 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 24 Packages 2024-04-15T15:18:33.876 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:18:33.876 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 228 M 2024-04-15T15:18:33.876 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-04-15T15:18:33.890 INFO:teuthology.orchestra.run.smithi012.stdout:(35/174): ceph-volume-19.0.0-2861.g9bf1c4ba.el8 2.8 MB/s | 264 kB 00:00 2024-04-15T15:18:34.015 INFO:teuthology.orchestra.run.smithi012.stdout:(36/174): cephadm-19.0.0-2861.g9bf1c4ba.el8.noa 5.9 MB/s | 762 kB 00:00 2024-04-15T15:18:34.107 INFO:teuthology.orchestra.run.smithi012.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 7.9 MB/s | 7.4 MB 00:00 2024-04-15T15:18:34.133 INFO:teuthology.orchestra.run.smithi012.stdout:(38/174): cephfs-top-19.0.0-2861.g9bf1c4ba.el8. 156 kB/s | 18 kB 00:00 2024-04-15T15:18:34.283 INFO:teuthology.orchestra.run.smithi012.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 662 kB 00:00 2024-04-15T15:18:34.325 INFO:teuthology.orchestra.run.smithi012.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-04-15T15:18:34.350 INFO:teuthology.orchestra.run.smithi012.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-04-15T15:18:34.375 INFO:teuthology.orchestra.run.smithi012.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-04-15T15:18:34.401 INFO:teuthology.orchestra.run.smithi012.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 635 kB/s | 32 kB 00:00 2024-04-15T15:18:34.527 INFO:teuthology.orchestra.run.smithi012.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-04-15T15:18:34.551 INFO:teuthology.orchestra.run.smithi012.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 115 kB/s | 17 kB 00:00 2024-04-15T15:18:34.576 INFO:teuthology.orchestra.run.smithi012.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 206 kB/s | 10 kB 00:00 2024-04-15T15:18:34.602 INFO:teuthology.orchestra.run.smithi012.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-15T15:18:34.627 INFO:teuthology.orchestra.run.smithi012.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-04-15T15:18:34.652 INFO:teuthology.orchestra.run.smithi012.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-04-15T15:18:34.678 INFO:teuthology.orchestra.run.smithi012.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-04-15T15:18:34.703 INFO:teuthology.orchestra.run.smithi012.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 931 kB/s | 46 kB 00:00 2024-04-15T15:18:34.729 INFO:teuthology.orchestra.run.smithi012.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-04-15T15:18:34.752 INFO:teuthology.orchestra.run.smithi055.stdout:(1/174): ceph-19.0.0-2861.g9bf1c4ba.el8.x86_64. 29 kB/s | 6.6 kB 00:00 2024-04-15T15:18:34.754 INFO:teuthology.orchestra.run.smithi012.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-04-15T15:18:34.779 INFO:teuthology.orchestra.run.smithi012.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-04-15T15:18:34.804 INFO:teuthology.orchestra.run.smithi012.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-04-15T15:18:34.905 INFO:teuthology.orchestra.run.smithi012.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-04-15T15:18:34.947 INFO:teuthology.orchestra.run.smithi012.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-04-15T15:18:34.980 INFO:teuthology.orchestra.run.smithi012.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 817 kB 00:00 2024-04-15T15:18:35.005 INFO:teuthology.orchestra.run.smithi012.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 132 kB 00:00 2024-04-15T15:18:35.069 INFO:teuthology.orchestra.run.smithi055.stdout:(2/174): ceph-base-19.0.0-2861.g9bf1c4ba.el8.x8 9.8 MB/s | 5.4 MB 00:00 2024-04-15T15:18:35.119 INFO:teuthology.orchestra.run.smithi055.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 2.8 MB/s | 145 kB 00:00 2024-04-15T15:18:35.278 INFO:teuthology.orchestra.run.smithi055.stdout:(4/174): ceph-mds-19.0.0-2861.g9bf1c4ba.el8.x86 15 MB/s | 2.4 MB 00:00 2024-04-15T15:18:35.407 INFO:teuthology.orchestra.run.smithi055.stdout:(5/174): ceph-mgr-19.0.0-2861.g9bf1c4ba.el8.x86 12 MB/s | 1.5 MB 00:00 2024-04-15T15:18:35.429 INFO:teuthology.orchestra.run.smithi055.stdout:(6/174): ceph-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 1.3 MB/s | 890 kB 00:00 2024-04-15T15:18:35.439 INFO:teuthology.orchestra.run.smithi012.stdout:(60/174): ceph-test-19.0.0-2861.g9bf1c4ba.el8.x 13 MB/s | 49 MB 00:03 2024-04-15T15:18:35.531 INFO:teuthology.orchestra.run.smithi012.stdout:(61/174): postgresql-server-13.14-1.el9.x86_64. 10 MB/s | 5.7 MB 00:00 2024-04-15T15:18:35.565 INFO:teuthology.orchestra.run.smithi012.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-04-15T15:18:35.590 INFO:teuthology.orchestra.run.smithi012.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 528 kB/s | 78 kB 00:00 2024-04-15T15:18:35.679 INFO:teuthology.orchestra.run.smithi055.stdout:(7/174): ceph-mon-19.0.0-2861.g9bf1c4ba.el8.x86 17 MB/s | 4.7 MB 00:00 2024-04-15T15:18:35.690 INFO:teuthology.orchestra.run.smithi012.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 37 MB/s | 5.8 MB 00:00 2024-04-15T15:18:35.707 INFO:teuthology.orchestra.run.smithi012.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 215 kB/s | 24 kB 00:00 2024-04-15T15:18:35.724 INFO:teuthology.orchestra.run.smithi012.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-04-15T15:18:35.749 INFO:teuthology.orchestra.run.smithi012.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-04-15T15:18:35.766 INFO:teuthology.orchestra.run.smithi012.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 763 kB/s | 32 kB 00:00 2024-04-15T15:18:35.850 INFO:teuthology.orchestra.run.smithi012.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-04-15T15:18:35.875 INFO:teuthology.orchestra.run.smithi012.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-04-15T15:18:35.900 INFO:teuthology.orchestra.run.smithi012.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 174 kB 00:00 2024-04-15T15:18:35.925 INFO:teuthology.orchestra.run.smithi012.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-04-15T15:18:35.951 INFO:teuthology.orchestra.run.smithi012.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-04-15T15:18:35.976 INFO:teuthology.orchestra.run.smithi012.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-04-15T15:18:36.001 INFO:teuthology.orchestra.run.smithi012.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-15T15:18:36.027 INFO:teuthology.orchestra.run.smithi012.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 954 kB/s | 47 kB 00:00 2024-04-15T15:18:36.052 INFO:teuthology.orchestra.run.smithi012.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-04-15T15:18:36.077 INFO:teuthology.orchestra.run.smithi012.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 568 kB/s | 43 kB 00:00 2024-04-15T15:18:36.219 INFO:teuthology.orchestra.run.smithi012.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 296 kB/s | 41 kB 00:00 2024-04-15T15:18:36.296 INFO:teuthology.orchestra.run.smithi055.stdout:(8/174): ceph-radosgw-19.0.0-2861.g9bf1c4ba.el8 16 MB/s | 10 MB 00:00 2024-04-15T15:18:36.338 INFO:teuthology.orchestra.run.smithi055.stdout:(9/174): ceph-selinux-19.0.0-2861.g9bf1c4ba.el8 609 kB/s | 25 kB 00:00 2024-04-15T15:18:36.378 INFO:teuthology.orchestra.run.smithi012.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-04-15T15:18:36.403 INFO:teuthology.orchestra.run.smithi012.stdout:(81/174): python3-toml-0.10.2-6.el9.noarch.rpm 120 kB/s | 39 kB 00:00 2024-04-15T15:18:36.428 INFO:teuthology.orchestra.run.smithi012.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2024-04-15T15:18:36.453 INFO:teuthology.orchestra.run.smithi012.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 933 kB/s | 46 kB 00:00 2024-04-15T15:18:36.487 INFO:teuthology.orchestra.run.smithi012.stdout:(84/174): qatlib-service-24.02.0-1.el9.x86_64.r 418 kB/s | 35 kB 00:00 2024-04-15T15:18:36.512 INFO:teuthology.orchestra.run.smithi012.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-04-15T15:18:36.538 INFO:teuthology.orchestra.run.smithi012.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 676 kB/s | 56 kB 00:00 2024-04-15T15:18:36.563 INFO:teuthology.orchestra.run.smithi012.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-04-15T15:18:36.588 INFO:teuthology.orchestra.run.smithi012.stdout:(88/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 833 kB/s | 63 kB 00:00 2024-04-15T15:18:36.613 INFO:teuthology.orchestra.run.smithi012.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-04-15T15:18:36.639 INFO:teuthology.orchestra.run.smithi012.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-04-15T15:18:36.664 INFO:teuthology.orchestra.run.smithi012.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-04-15T15:18:36.697 INFO:teuthology.orchestra.run.smithi012.stdout:(92/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-04-15T15:18:36.722 INFO:teuthology.orchestra.run.smithi012.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-04-15T15:18:36.748 INFO:teuthology.orchestra.run.smithi012.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-04-15T15:18:36.881 INFO:teuthology.orchestra.run.smithi012.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-15T15:18:36.907 INFO:teuthology.orchestra.run.smithi012.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-15T15:18:36.940 INFO:teuthology.orchestra.run.smithi012.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-15T15:18:36.974 INFO:teuthology.orchestra.run.smithi012.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-04-15T15:18:37.149 INFO:teuthology.orchestra.run.smithi012.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 89 kB/s | 21 kB 00:00 2024-04-15T15:18:37.166 INFO:teuthology.orchestra.run.smithi012.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 113 kB/s | 25 kB 00:00 2024-04-15T15:18:37.250 INFO:teuthology.orchestra.run.smithi012.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.4 MB/s | 245 kB 00:00 2024-04-15T15:18:37.275 INFO:teuthology.orchestra.run.smithi012.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 2.2 MB/s | 248 kB 00:00 2024-04-15T15:18:37.325 INFO:teuthology.orchestra.run.smithi012.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-15T15:18:37.401 INFO:teuthology.orchestra.run.smithi012.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 9.7 MB/s | 4.1 MB 00:00 2024-04-15T15:18:37.584 INFO:teuthology.orchestra.run.smithi012.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-04-15T15:18:38.568 INFO:teuthology.orchestra.run.smithi012.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 314 kB/s | 309 kB 00:00 2024-04-15T15:18:38.684 INFO:teuthology.orchestra.run.smithi012.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-04-15T15:18:38.726 INFO:teuthology.orchestra.run.smithi012.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 414 kB/s | 548 kB 00:01 2024-04-15T15:18:38.835 INFO:teuthology.orchestra.run.smithi012.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-15T15:18:38.952 INFO:teuthology.orchestra.run.smithi012.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-04-15T15:18:39.077 INFO:teuthology.orchestra.run.smithi012.stdout:(111/174): protobuf-compiler-3.14.0-13.el9.x86_ 493 kB/s | 863 kB 00:01 2024-04-15T15:18:39.161 INFO:teuthology.orchestra.run.smithi012.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 322 kB/s | 67 kB 00:00 2024-04-15T15:18:39.869 INFO:teuthology.orchestra.run.smithi012.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 191 kB/s | 151 kB 00:00 2024-04-15T15:18:39.980 INFO:teuthology.orchestra.run.smithi055.stdout:(10/174): ceph-osd-19.0.0-2861.g9bf1c4ba.el8.x8 3.7 MB/s | 17 MB 00:04 2024-04-15T15:18:40.295 INFO:teuthology.orchestra.run.smithi012.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 739 kB/s | 838 kB 00:01 2024-04-15T15:18:40.403 INFO:teuthology.orchestra.run.smithi012.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-15T15:18:40.405 INFO:teuthology.orchestra.run.smithi055.stdout:(11/174): ceph-test-19.0.0-2861.g9bf1c4ba.el8.x 12 MB/s | 49 MB 00:04 2024-04-15T15:18:40.430 INFO:teuthology.orchestra.run.smithi055.stdout:(12/174): cephfs-mirror-19.0.0-2861.g9bf1c4ba.e 499 kB/s | 224 kB 00:00 2024-04-15T15:18:40.480 INFO:teuthology.orchestra.run.smithi055.stdout:(13/174): cephfs-shell-19.0.0-2861.g9bf1c4ba.el 360 kB/s | 27 kB 00:00 2024-04-15T15:18:40.512 INFO:teuthology.orchestra.run.smithi012.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-15T15:18:40.522 INFO:teuthology.orchestra.run.smithi055.stdout:(14/174): libcephfs-devel-19.0.0-2861.g9bf1c4ba 345 kB/s | 32 kB 00:00 2024-04-15T15:18:40.620 INFO:teuthology.orchestra.run.smithi012.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-15T15:18:40.640 INFO:teuthology.orchestra.run.smithi055.stdout:(15/174): libcephsqlite-19.0.0-2861.g9bf1c4ba.e 1.4 MB/s | 164 kB 00:00 2024-04-15T15:18:40.646 INFO:teuthology.orchestra.run.smithi012.stdout:(118/174): python3-asyncssh-2.13.2-5.el9.noarch 706 kB/s | 548 kB 00:00 2024-04-15T15:18:40.724 INFO:teuthology.orchestra.run.smithi055.stdout:(16/174): librados-devel-19.0.0-2861.g9bf1c4ba. 1.5 MB/s | 127 kB 00:00 2024-04-15T15:18:40.766 INFO:teuthology.orchestra.run.smithi055.stdout:(17/174): libcephfs2-19.0.0-2861.g9bf1c4ba.el8. 2.6 MB/s | 747 kB 00:00 2024-04-15T15:18:40.888 INFO:teuthology.orchestra.run.smithi012.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 643 kB/s | 172 kB 00:00 2024-04-15T15:18:41.008 INFO:teuthology.orchestra.run.smithi055.stdout:(18/174): ceph-common-19.0.0-2861.g9bf1c4ba.el8 3.2 MB/s | 21 MB 00:06 2024-04-15T15:18:41.041 INFO:teuthology.orchestra.run.smithi055.stdout:(19/174): libradosstriper1-19.0.0-2861.g9bf1c4b 1.5 MB/s | 501 kB 00:00 2024-04-15T15:18:41.092 INFO:teuthology.orchestra.run.smithi055.stdout:(20/174): python3-ceph-argparse-19.0.0-2861.g9b 541 kB/s | 45 kB 00:00 2024-04-15T15:18:41.158 INFO:teuthology.orchestra.run.smithi055.stdout:(21/174): python3-ceph-common-19.0.0-2861.g9bf1 1.1 MB/s | 135 kB 00:00 2024-04-15T15:18:41.184 INFO:teuthology.orchestra.run.smithi055.stdout:(22/174): python3-cephfs-19.0.0-2861.g9bf1c4ba. 1.7 MB/s | 162 kB 00:00 2024-04-15T15:18:41.238 INFO:teuthology.orchestra.run.smithi012.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 606 kB/s | 358 kB 00:00 2024-04-15T15:18:41.346 INFO:teuthology.orchestra.run.smithi012.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-04-15T15:18:41.351 INFO:teuthology.orchestra.run.smithi055.stdout:(23/174): python3-rados-19.0.0-2861.g9bf1c4ba.e 1.6 MB/s | 320 kB 00:00 2024-04-15T15:18:41.384 INFO:teuthology.orchestra.run.smithi055.stdout:(24/174): python3-rbd-19.0.0-2861.g9bf1c4ba.el8 1.5 MB/s | 297 kB 00:00 2024-04-15T15:18:41.430 INFO:teuthology.orchestra.run.smithi012.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 559 kB/s | 303 kB 00:00 2024-04-15T15:18:41.493 INFO:teuthology.orchestra.run.smithi055.stdout:(25/174): python3-rgw-19.0.0-2861.g9bf1c4ba.el8 699 kB/s | 99 kB 00:00 2024-04-15T15:18:41.702 INFO:teuthology.orchestra.run.smithi055.stdout:(26/174): librgw2-19.0.0-2861.g9bf1c4ba.el8.x86 5.6 MB/s | 5.2 MB 00:00 2024-04-15T15:18:41.714 INFO:teuthology.orchestra.run.smithi012.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 552 kB/s | 203 kB 00:00 2024-04-15T15:18:41.752 INFO:teuthology.orchestra.run.smithi055.stdout:(27/174): rbd-nbd-19.0.0-2861.g9bf1c4ba.el8.x86 3.3 MB/s | 171 kB 00:00 2024-04-15T15:18:41.836 INFO:teuthology.orchestra.run.smithi055.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2861.g 411 kB/s | 34 kB 00:00 2024-04-15T15:18:41.961 INFO:teuthology.orchestra.run.smithi055.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2861.g9bf1c4b 1.1 MB/s | 142 kB 00:00 2024-04-15T15:18:41.987 INFO:teuthology.orchestra.run.smithi055.stdout:(30/174): rbd-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 142 kB/s | 85 kB 00:00 2024-04-15T15:18:41.997 INFO:teuthology.orchestra.run.smithi012.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 510 kB/s | 144 kB 00:00 2024-04-15T15:18:42.106 INFO:teuthology.orchestra.run.smithi012.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-04-15T15:18:42.214 INFO:teuthology.orchestra.run.smithi012.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-15T15:18:42.315 INFO:teuthology.orchestra.run.smithi012.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 232 kB/s | 23 kB 00:00 2024-04-15T15:18:42.403 INFO:teuthology.orchestra.run.smithi055.stdout:(31/174): rbd-mirror-19.0.0-2861.g9bf1c4ba.el8. 3.4 MB/s | 3.1 MB 00:00 2024-04-15T15:18:42.407 INFO:teuthology.orchestra.run.smithi012.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-04-15T15:18:42.499 INFO:teuthology.orchestra.run.smithi012.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-15T15:18:42.562 INFO:teuthology.orchestra.run.smithi055.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2861.g9b 1.5 MB/s | 249 kB 00:00 2024-04-15T15:18:42.621 INFO:teuthology.orchestra.run.smithi055.stdout:(33/174): ceph-mgr-rook-19.0.0-2861.g9bf1c4ba.e 846 kB/s | 49 kB 00:00 2024-04-15T15:18:42.696 INFO:teuthology.orchestra.run.smithi055.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2861.g9 219 kB/s | 16 kB 00:00 2024-04-15T15:18:42.838 INFO:teuthology.orchestra.run.smithi055.stdout:(35/174): ceph-volume-19.0.0-2861.g9bf1c4ba.el8 1.8 MB/s | 264 kB 00:00 2024-04-15T15:18:42.938 INFO:teuthology.orchestra.run.smithi055.stdout:(36/174): ceph-mgr-dashboard-19.0.0-2861.g9bf1c 3.6 MB/s | 3.5 MB 00:00 2024-04-15T15:18:43.005 INFO:teuthology.orchestra.run.smithi055.stdout:(37/174): cephfs-top-19.0.0-2861.g9bf1c4ba.el8. 273 kB/s | 18 kB 00:00 2024-04-15T15:18:43.089 INFO:teuthology.orchestra.run.smithi055.stdout:(38/174): cephadm-19.0.0-2861.g9bf1c4ba.el8.noa 3.0 MB/s | 762 kB 00:00 2024-04-15T15:18:43.231 INFO:teuthology.orchestra.run.smithi055.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 662 kB 00:00 2024-04-15T15:18:43.306 INFO:teuthology.orchestra.run.smithi055.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.1 MB/s | 920 kB 00:00 2024-04-15T15:18:43.332 INFO:teuthology.orchestra.run.smithi055.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 106 kB 00:00 2024-04-15T15:18:43.365 INFO:teuthology.orchestra.run.smithi055.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.1 MB/s | 307 kB 00:00 2024-04-15T15:18:43.374 INFO:teuthology.orchestra.run.smithi012.stdout:(130/174): libarrow-9.0.0-10.el9.x86_64.rpm 965 kB/s | 4.4 MB 00:04 2024-04-15T15:18:43.382 INFO:teuthology.orchestra.run.smithi055.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 631 kB/s | 32 kB 00:00 2024-04-15T15:18:43.483 INFO:teuthology.orchestra.run.smithi012.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 436 kB/s | 46 kB 00:00 2024-04-15T15:18:43.524 INFO:teuthology.orchestra.run.smithi055.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 121 kB/s | 17 kB 00:00 2024-04-15T15:18:43.583 INFO:teuthology.orchestra.run.smithi055.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-15T15:18:43.591 INFO:teuthology.orchestra.run.smithi012.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 728 kB/s | 79 kB 00:00 2024-04-15T15:18:43.608 INFO:teuthology.orchestra.run.smithi055.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 10 kB 00:00 2024-04-15T15:18:43.634 INFO:teuthology.orchestra.run.smithi055.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-15T15:18:43.659 INFO:teuthology.orchestra.run.smithi055.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-04-15T15:18:43.691 INFO:teuthology.orchestra.run.smithi012.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 577 kB/s | 58 kB 00:00 2024-04-15T15:18:43.701 INFO:teuthology.orchestra.run.smithi055.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-04-15T15:18:43.735 INFO:teuthology.orchestra.run.smithi055.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 578 kB 00:00 2024-04-15T15:18:43.760 INFO:teuthology.orchestra.run.smithi055.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 801 kB/s | 46 kB 00:00 2024-04-15T15:18:43.785 INFO:teuthology.orchestra.run.smithi055.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-04-15T15:18:43.808 INFO:teuthology.orchestra.run.smithi012.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-15T15:18:43.819 INFO:teuthology.orchestra.run.smithi055.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-04-15T15:18:43.842 INFO:teuthology.orchestra.run.smithi012.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 778 kB/s | 1.0 MB 00:01 2024-04-15T15:18:43.852 INFO:teuthology.orchestra.run.smithi055.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-04-15T15:18:43.878 INFO:teuthology.orchestra.run.smithi055.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 580 kB/s | 34 kB 00:00 2024-04-15T15:18:43.917 INFO:teuthology.orchestra.run.smithi012.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-15T15:18:44.026 INFO:teuthology.orchestra.run.smithi012.stdout:(137/174): python3-grpcio-1.46.7-10.el9.x86_64. 804 kB/s | 2.0 MB 00:02 2024-04-15T15:18:44.043 INFO:teuthology.orchestra.run.smithi012.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2024-04-15T15:18:44.068 INFO:teuthology.orchestra.run.smithi012.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 398 kB/s | 90 kB 00:00 2024-04-15T15:18:44.127 INFO:teuthology.orchestra.run.smithi012.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 305 kB/s | 31 kB 00:00 2024-04-15T15:18:44.136 INFO:teuthology.orchestra.run.smithi055.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-04-15T15:18:44.144 INFO:teuthology.orchestra.run.smithi012.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-15T15:18:44.195 INFO:teuthology.orchestra.run.smithi055.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 4.8 MB/s | 1.5 MB 00:00 2024-04-15T15:18:44.219 INFO:teuthology.orchestra.run.smithi012.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-04-15T15:18:44.229 INFO:teuthology.orchestra.run.smithi055.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 817 kB 00:00 2024-04-15T15:18:44.244 INFO:teuthology.orchestra.run.smithi012.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-04-15T15:18:44.254 INFO:teuthology.orchestra.run.smithi055.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 132 kB 00:00 2024-04-15T15:18:44.295 INFO:teuthology.orchestra.run.smithi012.stdout:(144/174): python3-rsa-4.9-2.el9.noarch.rpm 261 kB/s | 59 kB 00:00 2024-04-15T15:18:44.378 INFO:teuthology.orchestra.run.smithi012.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 671 kB/s | 90 kB 00:00 2024-04-15T15:18:44.495 INFO:teuthology.orchestra.run.smithi012.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-04-15T15:18:44.554 INFO:teuthology.orchestra.run.smithi055.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 18 MB/s | 5.7 MB 00:00 2024-04-15T15:18:44.588 INFO:teuthology.orchestra.run.smithi055.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-04-15T15:18:44.604 INFO:teuthology.orchestra.run.smithi012.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-15T15:18:44.613 INFO:teuthology.orchestra.run.smithi055.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-04-15T15:18:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:(148/174): python3-webob-1.8.7-6.el9.noarch.rpm 560 kB/s | 230 kB 00:00 2024-04-15T15:18:44.755 INFO:teuthology.orchestra.run.smithi055.stdout:(63/174): ceph-mgr-diskprediction-local-19.0.0- 2.7 MB/s | 7.4 MB 00:02 2024-04-15T15:18:44.864 INFO:teuthology.orchestra.run.smithi055.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-04-15T15:18:44.881 INFO:teuthology.orchestra.run.smithi055.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 92 kB/s | 24 kB 00:00 2024-04-15T15:18:44.896 INFO:teuthology.orchestra.run.smithi012.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-04-15T15:18:44.898 INFO:teuthology.orchestra.run.smithi055.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 130 kB/s | 18 kB 00:00 2024-04-15T15:18:44.915 INFO:teuthology.orchestra.run.smithi055.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 940 kB/s | 32 kB 00:00 2024-04-15T15:18:44.940 INFO:teuthology.orchestra.run.smithi055.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-04-15T15:18:44.988 INFO:teuthology.orchestra.run.smithi012.stdout:(150/174): python3-werkzeug-2.0.3-3.el9.1.noarc 616 kB/s | 427 kB 00:00 2024-04-15T15:18:45.107 INFO:teuthology.orchestra.run.smithi055.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-04-15T15:18:45.133 INFO:teuthology.orchestra.run.smithi055.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-04-15T15:18:45.166 INFO:teuthology.orchestra.run.smithi055.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 780 kB/s | 174 kB 00:00 2024-04-15T15:18:45.184 INFO:teuthology.orchestra.run.smithi055.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-04-15T15:18:45.197 INFO:teuthology.orchestra.run.smithi012.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-04-15T15:18:45.200 INFO:teuthology.orchestra.run.smithi055.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-04-15T15:18:45.225 INFO:teuthology.orchestra.run.smithi055.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-04-15T15:18:45.247 INFO:teuthology.orchestra.run.smithi012.stdout:(152/174): librados2-19.0.0-2861.g9bf1c4ba.el8. 5.5 MB/s | 3.4 MB 00:00 2024-04-15T15:18:45.251 INFO:teuthology.orchestra.run.smithi055.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-15T15:18:45.276 INFO:teuthology.orchestra.run.smithi055.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-04-15T15:18:45.289 INFO:teuthology.orchestra.run.smithi012.stdout:(153/174): NetworkManager-libnm-1.47.4-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-04-15T15:18:45.301 INFO:teuthology.orchestra.run.smithi055.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 954 kB/s | 47 kB 00:00 2024-04-15T15:18:45.314 INFO:teuthology.orchestra.run.smithi012.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 632 kB/s | 41 kB 00:00 2024-04-15T15:18:45.326 INFO:teuthology.orchestra.run.smithi055.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2024-04-15T15:18:45.340 INFO:teuthology.orchestra.run.smithi012.stdout:(155/174): NetworkManager-tui-1.47.4-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-04-15T15:18:45.364 INFO:teuthology.orchestra.run.smithi012.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-15T15:18:45.415 INFO:teuthology.orchestra.run.smithi012.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 957 kB/s | 71 kB 00:00 2024-04-15T15:18:45.449 INFO:teuthology.orchestra.run.smithi012.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 436 kB/s | 36 kB 00:00 2024-04-15T15:18:45.482 INFO:teuthology.orchestra.run.smithi012.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-04-15T15:18:45.516 INFO:teuthology.orchestra.run.smithi012.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 693 kB/s | 23 kB 00:00 2024-04-15T15:18:45.549 INFO:teuthology.orchestra.run.smithi012.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-04-15T15:18:45.568 INFO:teuthology.orchestra.run.smithi055.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 162 kB/s | 39 kB 00:00 2024-04-15T15:18:45.574 INFO:teuthology.orchestra.run.smithi012.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-04-15T15:18:45.594 INFO:teuthology.orchestra.run.smithi055.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 155 kB/s | 41 kB 00:00 2024-04-15T15:18:45.600 INFO:teuthology.orchestra.run.smithi012.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-04-15T15:18:45.625 INFO:teuthology.orchestra.run.smithi012.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-04-15T15:18:45.642 INFO:teuthology.orchestra.run.smithi012.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-04-15T15:18:45.692 INFO:teuthology.orchestra.run.smithi012.stdout:(166/174): librbd1-19.0.0-2861.g9bf1c4ba.el8.x8 3.9 MB/s | 3.1 MB 00:00 2024-04-15T15:18:45.709 INFO:teuthology.orchestra.run.smithi012.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 342 kB/s | 29 kB 00:00 2024-04-15T15:18:45.760 INFO:teuthology.orchestra.run.smithi012.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-04-15T15:18:45.769 INFO:teuthology.orchestra.run.smithi055.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-04-15T15:18:45.794 INFO:teuthology.orchestra.run.smithi055.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 977 kB/s | 220 kB 00:00 2024-04-15T15:18:45.802 INFO:teuthology.orchestra.run.smithi012.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 11 MB/s | 465 kB 00:00 2024-04-15T15:18:45.819 INFO:teuthology.orchestra.run.smithi055.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 156 kB/s | 35 kB 00:00 2024-04-15T15:18:45.844 INFO:teuthology.orchestra.run.smithi012.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.3 MB 00:00 2024-04-15T15:18:45.845 INFO:teuthology.orchestra.run.smithi055.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 621 kB/s | 46 kB 00:00 2024-04-15T15:18:45.860 INFO:teuthology.orchestra.run.smithi012.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 534 kB/s | 31 kB 00:00 2024-04-15T15:18:45.870 INFO:teuthology.orchestra.run.smithi055.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-04-15T15:18:45.895 INFO:teuthology.orchestra.run.smithi055.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 751 kB/s | 56 kB 00:00 2024-04-15T15:18:45.903 INFO:teuthology.orchestra.run.smithi012.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-04-15T15:18:45.921 INFO:teuthology.orchestra.run.smithi055.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-04-15T15:18:45.946 INFO:teuthology.orchestra.run.smithi055.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-04-15T15:18:45.953 INFO:teuthology.orchestra.run.smithi012.stdout:(173/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.3 MB/s | 2.2 MB 00:00 2024-04-15T15:18:45.971 INFO:teuthology.orchestra.run.smithi055.stdout:(89/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 585 kB/s | 44 kB 00:00 2024-04-15T15:18:45.996 INFO:teuthology.orchestra.run.smithi055.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-04-15T15:18:46.039 INFO:teuthology.orchestra.run.smithi055.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-04-15T15:18:46.064 INFO:teuthology.orchestra.run.smithi055.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-04-15T15:18:46.089 INFO:teuthology.orchestra.run.smithi055.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-04-15T15:18:46.095 INFO:teuthology.orchestra.run.smithi012.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 318 kB/s | 74 kB 00:00 2024-04-15T15:18:46.100 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:18:46.100 INFO:teuthology.orchestra.run.smithi012.stdout:Total 13 MB/s | 228 MB 00:17 2024-04-15T15:18:46.114 INFO:teuthology.orchestra.run.smithi055.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-04-15T15:18:46.281 INFO:teuthology.orchestra.run.smithi055.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 978 kB/s | 187 kB 00:00 2024-04-15T15:18:46.306 INFO:teuthology.orchestra.run.smithi055.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-04-15T15:18:46.474 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-04-15T15:18:46.507 INFO:teuthology.orchestra.run.smithi055.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 107 kB/s | 21 kB 00:00 2024-04-15T15:18:46.532 INFO:teuthology.orchestra.run.smithi055.stdout:(98/174): python3-requests-2.25.1-8.el9.noarch. 497 kB/s | 125 kB 00:00 2024-04-15T15:18:46.558 INFO:teuthology.orchestra.run.smithi055.stdout:(99/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-15T15:18:46.574 INFO:teuthology.orchestra.run.smithi055.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 380 kB/s | 25 kB 00:00 2024-04-15T15:18:46.667 INFO:teuthology.orchestra.run.smithi055.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.2 MB/s | 245 kB 00:00 2024-04-15T15:18:46.717 INFO:teuthology.orchestra.run.smithi055.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.7 MB/s | 248 kB 00:00 2024-04-15T15:18:46.742 INFO:teuthology.orchestra.run.smithi055.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-15T15:18:46.771 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-04-15T15:18:46.771 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-04-15T15:18:46.826 INFO:teuthology.orchestra.run.smithi055.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-04-15T15:18:46.968 INFO:teuthology.orchestra.run.smithi055.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-04-15T15:18:47.643 INFO:teuthology.orchestra.run.smithi055.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.7 MB/s | 4.1 MB 00:01 2024-04-15T15:18:47.885 INFO:teuthology.orchestra.run.smithi055.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 337 kB/s | 309 kB 00:00 2024-04-15T15:18:48.036 INFO:teuthology.orchestra.run.smithi055.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 453 kB/s | 548 kB 00:01 2024-04-15T15:18:48.086 INFO:teuthology.orchestra.run.smithi055.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 44 kB/s | 19 kB 00:00 2024-04-15T15:18:48.128 INFO:teuthology.orchestra.run.smithi055.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 282 kB/s | 26 kB 00:00 2024-04-15T15:18:48.270 INFO:teuthology.orchestra.run.smithi055.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 264 kB/s | 49 kB 00:00 2024-04-15T15:18:48.312 INFO:teuthology.orchestra.run.smithi055.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-04-15T15:18:48.537 INFO:teuthology.orchestra.run.smithi055.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 566 kB/s | 151 kB 00:00 2024-04-15T15:18:48.874 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-04-15T15:18:48.874 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-04-15T15:18:49.279 INFO:teuthology.orchestra.run.smithi055.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 739 kB/s | 548 kB 00:00 2024-04-15T15:18:49.389 INFO:teuthology.orchestra.run.smithi055.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 779 kB/s | 838 kB 00:01 2024-04-15T15:18:49.413 INFO:teuthology.orchestra.run.smithi055.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 325 kB/s | 43 kB 00:00 2024-04-15T15:18:49.488 INFO:teuthology.orchestra.run.smithi055.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-04-15T15:18:49.514 INFO:teuthology.orchestra.run.smithi055.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-15T15:18:49.755 INFO:teuthology.orchestra.run.smithi055.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 643 kB/s | 172 kB 00:00 2024-04-15T15:18:50.023 INFO:teuthology.orchestra.run.smithi055.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 705 kB/s | 358 kB 00:00 2024-04-15T15:18:50.131 INFO:teuthology.orchestra.run.smithi055.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-04-15T15:18:50.277 INFO:teuthology.orchestra.run.smithi055.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 581 kB/s | 303 kB 00:00 2024-04-15T15:18:50.490 INFO:teuthology.orchestra.run.smithi055.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 565 kB/s | 203 kB 00:00 2024-04-15T15:18:50.749 INFO:teuthology.orchestra.run.smithi055.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 559 kB/s | 144 kB 00:00 2024-04-15T15:18:50.849 INFO:teuthology.orchestra.run.smithi055.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-15T15:18:50.949 INFO:teuthology.orchestra.run.smithi055.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-04-15T15:18:51.050 INFO:teuthology.orchestra.run.smithi055.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-15T15:18:51.150 INFO:teuthology.orchestra.run.smithi055.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-15T15:18:51.250 INFO:teuthology.orchestra.run.smithi055.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-15T15:18:52.584 INFO:teuthology.orchestra.run.smithi055.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 784 kB/s | 1.0 MB 00:01 2024-04-15T15:18:52.684 INFO:teuthology.orchestra.run.smithi055.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-04-15T15:18:52.885 INFO:teuthology.orchestra.run.smithi055.stdout:(132/174): python3-grpcio-1.46.7-10.el9.x86_64. 800 kB/s | 2.0 MB 00:02 2024-04-15T15:18:52.912 INFO:teuthology.orchestra.run.smithi055.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 346 kB/s | 79 kB 00:00 2024-04-15T15:18:53.060 INFO:teuthology.orchestra.run.smithi055.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 329 kB/s | 58 kB 00:00 2024-04-15T15:18:53.161 INFO:teuthology.orchestra.run.smithi055.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-15T15:18:53.344 INFO:teuthology.orchestra.run.smithi055.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 631 kB/s | 272 kB 00:00 2024-04-15T15:18:53.369 INFO:teuthology.orchestra.run.smithi055.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 431 kB/s | 90 kB 00:00 2024-04-15T15:18:53.445 INFO:teuthology.orchestra.run.smithi055.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-04-15T15:18:53.520 INFO:teuthology.orchestra.run.smithi055.stdout:(139/174): libarrow-9.0.0-10.el9.x86_64.rpm 803 kB/s | 4.4 MB 00:05 2024-04-15T15:18:53.537 INFO:teuthology.orchestra.run.smithi055.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 184 kB/s | 31 kB 00:00 2024-04-15T15:18:53.637 INFO:teuthology.orchestra.run.smithi055.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 358 kB/s | 36 kB 00:00 2024-04-15T15:18:53.721 INFO:teuthology.orchestra.run.smithi055.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 684 kB/s | 188 kB 00:00 2024-04-15T15:18:53.738 INFO:teuthology.orchestra.run.smithi055.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-04-15T15:18:53.755 INFO:teuthology.orchestra.run.smithi055.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 435 kB/s | 51 kB 00:00 2024-04-15T15:18:53.922 INFO:teuthology.orchestra.run.smithi055.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 489 kB/s | 90 kB 00:00 2024-04-15T15:18:54.014 INFO:teuthology.orchestra.run.smithi055.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 218 kB/s | 20 kB 00:00 2024-04-15T15:18:54.064 INFO:teuthology.orchestra.run.smithi055.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 670 kB/s | 230 kB 00:00 2024-04-15T15:18:54.273 INFO:teuthology.orchestra.run.smithi055.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 740 kB/s | 192 kB 00:00 2024-04-15T15:18:54.382 INFO:teuthology.orchestra.run.smithi055.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 682 kB/s | 427 kB 00:00 2024-04-15T15:18:54.807 INFO:teuthology.orchestra.run.smithi055.stdout:(150/174): librbd1-19.0.0-2861.g9bf1c4ba.el8.x8 7.4 MB/s | 3.1 MB 00:00 2024-04-15T15:18:54.907 INFO:teuthology.orchestra.run.smithi055.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-04-15T15:18:54.982 INFO:teuthology.orchestra.run.smithi055.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-04-15T15:18:55.041 INFO:teuthology.orchestra.run.smithi055.stdout:(153/174): librados2-19.0.0-2861.g9bf1c4ba.el8. 4.4 MB/s | 3.4 MB 00:00 2024-04-15T15:18:55.066 INFO:teuthology.orchestra.run.smithi055.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 487 kB/s | 41 kB 00:00 2024-04-15T15:18:55.092 INFO:teuthology.orchestra.run.smithi055.stdout:(155/174): NetworkManager-tui-1.47.4-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-04-15T15:18:55.117 INFO:teuthology.orchestra.run.smithi055.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-15T15:18:55.184 INFO:teuthology.orchestra.run.smithi055.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 548 kB/s | 36 kB 00:00 2024-04-15T15:18:55.210 INFO:teuthology.orchestra.run.smithi055.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 611 kB/s | 71 kB 00:00 2024-04-15T15:18:55.260 INFO:teuthology.orchestra.run.smithi055.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-04-15T15:18:55.310 INFO:teuthology.orchestra.run.smithi055.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 461 kB/s | 23 kB 00:00 2024-04-15T15:18:55.336 INFO:teuthology.orchestra.run.smithi055.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-04-15T15:18:55.352 INFO:teuthology.orchestra.run.smithi055.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-04-15T15:18:55.394 INFO:teuthology.orchestra.run.smithi055.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-04-15T15:18:55.411 INFO:teuthology.orchestra.run.smithi055.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-04-15T15:18:55.437 INFO:teuthology.orchestra.run.smithi055.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-04-15T15:18:55.454 INFO:teuthology.orchestra.run.smithi055.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 683 kB/s | 29 kB 00:00 2024-04-15T15:18:55.646 INFO:teuthology.orchestra.run.smithi055.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 5.8 MB/s | 1.2 MB 00:00 2024-04-15T15:18:55.804 INFO:teuthology.orchestra.run.smithi055.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.2 MB/s | 2.2 MB 00:00 2024-04-15T15:18:55.913 INFO:teuthology.orchestra.run.smithi055.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.6 MB/s | 2.3 MB 00:00 2024-04-15T15:18:55.963 INFO:teuthology.orchestra.run.smithi055.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 2.9 MB/s | 465 kB 00:00 2024-04-15T15:18:55.980 INFO:teuthology.orchestra.run.smithi055.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 466 kB/s | 31 kB 00:00 2024-04-15T15:18:56.031 INFO:teuthology.orchestra.run.smithi055.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.5 MB/s | 74 kB 00:00 2024-04-15T15:18:56.178 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-04-15T15:18:56.181 INFO:teuthology.orchestra.run.smithi055.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.9 MB/s | 1.1 MB 00:00 2024-04-15T15:18:56.214 INFO:teuthology.orchestra.run.smithi055.stdout:(174/174): thrift-0.15.0-2.el9.x86_64.rpm 753 kB/s | 1.6 MB 00:02 2024-04-15T15:18:56.221 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:18:56.221 INFO:teuthology.orchestra.run.smithi055.stdout:Total 10 MB/s | 228 MB 00:22 2024-04-15T15:18:56.274 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-15T15:18:56.308 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-15T15:18:56.346 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:18:56.356 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:18:56.397 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-15T15:18:56.451 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-15T15:18:56.492 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-15T15:18:56.534 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-15T15:18:56.579 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-15T15:18:56.594 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-04-15T15:18:56.891 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-04-15T15:18:56.891 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-04-15T15:18:56.933 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-15T15:18:56.987 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:18:57.108 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:18:57.138 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:18:57.230 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:18:57.259 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-15T15:18:57.494 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-15T15:18:57.556 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-15T15:18:57.585 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:18:57.682 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:18:57.711 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:18:57.784 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:18:57.823 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-15T15:18:57.846 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:18:57.862 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:18:58.048 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:18:58.078 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:18:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-15T15:18:58.296 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-15T15:18:58.414 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:18:58.458 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:18:58.561 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:18:58.609 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:18:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:18:58.952 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-15T15:18:59.001 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-04-15T15:18:59.001 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-04-15T15:18:59.046 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-15T15:18:59.089 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-15T15:18:59.120 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-15T15:18:59.177 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-15T15:18:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-15T15:18:59.308 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-15T15:18:59.346 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-15T15:18:59.420 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-15T15:18:59.461 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-15T15:18:59.504 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-15T15:18:59.537 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-15T15:18:59.564 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-15T15:18:59.637 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-15T15:18:59.669 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 39/198 2024-04-15T15:18:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 40/198 2024-04-15T15:18:59.741 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 41/198 2024-04-15T15:18:59.814 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-15T15:18:59.856 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-15T15:18:59.933 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-15T15:19:00.078 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-15T15:19:00.134 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-15T15:19:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-15T15:19:00.772 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-15T15:19:00.868 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-15T15:19:00.906 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:19:00.915 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:19:00.955 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-15T15:19:01.004 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-15T15:19:01.042 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 53/198 2024-04-15T15:19:01.080 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-15T15:19:01.338 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-15T15:19:01.515 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-15T15:19:01.599 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-15T15:19:01.654 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-15T15:19:02.052 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-15T15:19:02.229 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-15T15:19:02.285 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-15T15:19:02.358 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-15T15:19:02.387 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-15T15:19:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-15T15:19:02.463 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-15T15:19:02.498 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-15T15:19:02.542 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-15T15:19:02.584 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-15T15:19:02.723 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-15T15:19:02.773 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-15T15:19:02.802 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-15T15:19:02.869 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-15T15:19:02.975 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-15T15:19:03.011 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-15T15:19:03.050 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-15T15:19:03.076 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-15T15:19:03.101 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-15T15:19:03.187 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-15T15:19:03.316 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-15T15:19:03.422 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-15T15:19:03.484 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-15T15:19:03.541 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-15T15:19:03.598 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-15T15:19:03.792 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-15T15:19:03.839 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-15T15:19:03.983 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-15T15:19:04.020 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-15T15:19:04.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-15T15:19:04.082 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:19:04.110 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-15T15:19:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-15T15:19:04.184 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-15T15:19:04.216 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-15T15:19:04.246 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-15T15:19:04.324 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-15T15:19:04.389 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-15T15:19:04.423 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-15T15:19:04.465 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-15T15:19:04.499 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-15T15:19:04.592 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-15T15:19:04.689 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-15T15:19:04.738 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-15T15:19:04.769 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-15T15:19:04.817 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-15T15:19:04.927 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-15T15:19:04.980 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-15T15:19:05.746 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-15T15:19:05.793 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-15T15:19:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-15T15:19:05.876 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-15T15:19:05.967 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-15T15:19:06.010 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-15T15:19:06.110 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-04-15T15:19:06.195 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-15T15:19:06.202 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-15T15:19:06.216 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-15T15:19:06.246 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-15T15:19:06.284 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:19:06.293 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:19:06.303 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-15T15:19:06.339 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-15T15:19:06.389 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-15T15:19:06.430 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-15T15:19:06.463 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-15T15:19:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-15T15:19:06.863 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-15T15:19:06.919 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:19:06.974 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-15T15:19:07.068 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:19:07.075 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-15T15:19:07.096 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:19:07.172 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:19:07.200 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:19:07.238 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-15T15:19:07.419 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-15T15:19:07.475 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-15T15:19:07.502 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:19:07.607 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:19:07.636 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:19:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:19:07.748 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-15T15:19:07.771 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:19:07.787 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:19:07.964 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:19:07.995 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:19:08.034 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-15T15:19:08.213 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-15T15:19:08.343 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:19:08.362 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-15T15:19:08.394 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:19:08.434 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-15T15:19:08.464 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-15T15:19:08.496 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-15T15:19:08.505 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:19:08.523 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-15T15:19:08.546 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:19:08.767 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:19:08.851 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-15T15:19:08.880 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-15T15:19:08.886 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:19:08.964 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:19:08.991 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 125/198 2024-04-15T15:19:08.992 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-15T15:19:09.026 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-15T15:19:09.030 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-15T15:19:09.053 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-15T15:19:09.097 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-15T15:19:09.174 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-15T15:19:09.227 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-15T15:19:09.264 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-15T15:19:09.339 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-15T15:19:09.379 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-15T15:19:09.423 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-15T15:19:09.456 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-15T15:19:09.483 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-15T15:19:09.519 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-15T15:19:09.547 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:19:09.555 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-15T15:19:09.590 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 39/198 2024-04-15T15:19:09.610 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:19:09.632 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 40/198 2024-04-15T15:19:09.632 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 129/198 2024-04-15T15:19:09.660 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 41/198 2024-04-15T15:19:09.731 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-15T15:19:09.783 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-15T15:19:09.868 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-15T15:19:10.006 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-15T15:19:10.062 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-15T15:19:10.614 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-15T15:19:10.699 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-15T15:19:10.775 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-15T15:19:10.817 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:19:10.826 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:19:10.858 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-15T15:19:10.907 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-15T15:19:10.944 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 53/198 2024-04-15T15:19:10.975 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-15T15:19:11.254 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-15T15:19:11.443 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-15T15:19:11.534 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-15T15:19:11.579 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-15T15:19:11.727 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-15T15:19:11.782 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-15T15:19:11.838 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-15T15:19:11.911 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-15T15:19:11.940 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-15T15:19:11.959 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:19:11.977 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-15T15:19:11.995 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:19:12.024 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-15T15:19:12.068 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-15T15:19:12.112 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-15T15:19:12.154 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-15T15:19:12.293 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-15T15:19:12.342 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-15T15:19:12.373 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-15T15:19:12.414 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-15T15:19:12.511 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-15T15:19:12.555 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-15T15:19:12.564 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:19:12.596 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-15T15:19:12.598 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:19:12.621 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-15T15:19:12.645 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-15T15:19:12.698 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:19:12.698 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T15:19:12.699 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T15:19:12.699 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:12.727 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-15T15:19:12.741 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:19:12.859 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-15T15:19:12.967 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-15T15:19:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-15T15:19:13.063 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-15T15:19:13.118 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-15T15:19:13.309 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-15T15:19:13.367 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-15T15:19:13.509 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-15T15:19:13.549 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-15T15:19:13.580 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-15T15:19:13.611 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:19:13.639 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-15T15:19:13.673 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-15T15:19:13.729 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-15T15:19:13.761 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-15T15:19:13.791 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-15T15:19:13.869 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-15T15:19:13.901 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-15T15:19:13.934 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-15T15:19:13.968 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-15T15:19:14.002 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-15T15:19:14.094 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-15T15:19:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-15T15:19:14.250 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-15T15:19:14.281 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-15T15:19:14.320 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-15T15:19:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-15T15:19:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-15T15:19:15.242 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-15T15:19:15.288 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-15T15:19:15.322 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-15T15:19:15.370 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-15T15:19:15.460 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-15T15:19:15.498 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-15T15:19:15.682 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-15T15:19:15.712 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-15T15:19:15.792 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-15T15:19:16.477 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-15T15:19:16.622 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-15T15:19:17.904 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-15T15:19:17.972 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-15T15:19:18.003 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-15T15:19:18.034 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-15T15:19:18.061 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-15T15:19:18.390 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-15T15:19:18.417 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:19:18.491 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:19:18.522 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 125/198 2024-04-15T15:19:18.558 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-15T15:19:19.050 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-15T15:19:19.078 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:19:19.149 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:19:19.179 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 129/198 2024-04-15T15:19:21.459 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:19:21.528 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:19:22.099 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:19:22.139 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:19:22.229 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:19:22.229 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T15:19:22.229 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T15:19:22.229 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:22.265 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:19:25.223 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:19:25.224 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-04-15T15:19:25.224 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-04-15T15:19:25.224 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-04-15T15:19:25.224 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-04-15T15:19:25.224 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-04-15T15:19:25.224 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2024-04-15T15:19:25.224 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-04-15T15:19:25.224 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:25.403 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:19:25.418 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:19:25.419 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:25.419 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-15T15:19:25.419 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:26.963 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:19:26.989 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:19:26.989 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:26.989 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T15:19:26.989 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:19:26.989 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:19:26.989 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:27.248 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:19:27.271 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:19:27.274 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:27.274 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T15:19:27.274 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:19:27.274 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:19:27.274 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:27.741 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:19:27.765 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:19:27.765 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:27.765 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T15:19:27.765 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:19:27.765 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:19:27.765 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:27.859 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-15T15:19:28.271 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:19:28.299 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:19:28.332 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 139/198 2024-04-15T15:19:28.395 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 140/198 2024-04-15T15:19:28.422 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:19:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:19:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:19:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:19:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:19:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:19:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 144/198 2024-04-15T15:19:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:19:30.909 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:19:30.909 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:30.909 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T15:19:30.909 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:19:30.910 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:19:30.910 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:19:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:19:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 147/198 2024-04-15T15:19:31.944 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-15T15:19:31.971 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:19:31.994 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:19:31.994 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:31.994 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T15:19:31.994 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:19:31.994 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:19:31.994 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:19:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:19:32.060 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:32.061 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T15:19:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:32.114 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:19:32.137 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:19:32.137 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:32.137 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-15T15:19:32.137 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:19:32.137 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:19:32.137 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:32.457 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:19:32.480 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:19:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T15:19:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:19:32.480 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:19:32.480 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:34.734 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:19:34.734 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2024-04-15T15:19:34.734 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2024-04-15T15:19:34.734 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2024-04-15T15:19:34.734 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2024-04-15T15:19:34.734 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2024-04-15T15:19:34.735 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /root 2024-04-15T15:19:34.735 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2024-04-15T15:19:34.735 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:34.894 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:19:34.909 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:19:34.909 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:34.909 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-15T15:19:34.909 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:36.437 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:19:36.464 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:19:36.464 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:36.464 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T15:19:36.465 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:19:36.465 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:19:36.465 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:36.705 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:19:36.729 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:19:36.729 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:36.729 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T15:19:36.729 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:19:36.729 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:19:36.729 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:37.191 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:19:37.214 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:19:37.214 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:37.214 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T15:19:37.214 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:19:37.214 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:19:37.214 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:37.302 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:19:37.309 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-15T15:19:37.344 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 154/198 2024-04-15T15:19:37.373 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 155/198 2024-04-15T15:19:37.413 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-15T15:19:37.599 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 157/198 2024-04-15T15:19:37.688 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:19:37.716 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:19:37.721 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-15T15:19:37.740 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 139/198 2024-04-15T15:19:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 159/198 2024-04-15T15:19:37.803 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 140/198 2024-04-15T15:19:37.823 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-15T15:19:37.831 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:19:37.874 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-15T15:19:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-15T15:19:37.941 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 163/198 2024-04-15T15:19:37.981 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-15T15:19:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-15T15:19:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-15T15:19:38.123 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-15T15:19:38.160 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-15T15:19:38.690 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:19:38.724 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:19:38.743 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:19:38.937 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:19:38.974 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:19:39.618 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 170/198 2024-04-15T15:19:39.696 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-15T15:19:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-15T15:19:39.833 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-15T15:19:39.874 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-15T15:19:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-15T15:19:39.925 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-15T15:19:39.951 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-15T15:19:39.952 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:19:39.995 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:19:40.016 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:19:40.039 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:19:40.069 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-15T15:19:40.069 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:19:40.125 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:19:40.153 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:19:40.197 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-15T15:19:40.246 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-15T15:19:40.271 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-15T15:19:40.287 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 144/198 2024-04-15T15:19:40.296 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-15T15:19:40.326 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:19:40.332 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-15T15:19:40.350 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:19:40.350 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:40.351 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T15:19:40.351 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:19:40.351 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:19:40.351 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:40.377 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-15T15:19:40.402 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:19:40.409 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-15T15:19:40.414 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:19:40.435 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-15T15:19:40.464 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 147/198 2024-04-15T15:19:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-15T15:19:40.502 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:19:40.527 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-15T15:19:40.533 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:19:40.534 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:19:40.721 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:19:40.722 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:19:40.804 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:19:40.836 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:19:40.862 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-15T15:19:40.889 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-15T15:19:40.913 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-15T15:19:40.982 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:19:41.395 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-15T15:19:41.429 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:19:41.452 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:19:41.453 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:41.453 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T15:19:41.453 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:19:41.453 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:19:41.453 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:41.496 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:19:41.519 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:19:41.519 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:41.519 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T15:19:41.519 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:41.581 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:19:41.603 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:19:41.603 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:41.603 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-15T15:19:41.603 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:19:41.604 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:19:41.604 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:41.915 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:19:41.938 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:19:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:19:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T15:19:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:19:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:19:41.938 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 1/198 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2/198 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 3/198 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 4/198 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 5/198 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 6/198 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 7/198 2024-04-15T15:19:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 8/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 9/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 13/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 14/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 15/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 17/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 18/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 19/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 20/198 2024-04-15T15:19:46.332 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 21/198 2024-04-15T15:19:46.333 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 22/198 2024-04-15T15:19:46.333 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 23/198 2024-04-15T15:19:46.333 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 24/198 2024-04-15T15:19:46.333 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 25/198 2024-04-15T15:19:46.333 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 26/198 2024-04-15T15:19:46.333 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 27/198 2024-04-15T15:19:46.333 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 28/198 2024-04-15T15:19:46.333 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 29/198 2024-04-15T15:19:46.349 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 30/198 2024-04-15T15:19:46.349 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 31/198 2024-04-15T15:19:46.349 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 32/198 2024-04-15T15:19:46.349 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 33/198 2024-04-15T15:19:46.349 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 34/198 2024-04-15T15:19:46.349 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 35/198 2024-04-15T15:19:46.349 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 36/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 37/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 38/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 39/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-15T15:19:46.350 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-15T15:19:46.351 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-15T15:19:46.352 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-15T15:19:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-15T15:19:46.354 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-15T15:19:46.355 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-15T15:19:46.356 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-15T15:19:46.357 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-15T15:19:46.358 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-15T15:19:46.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-15T15:19:46.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-15T15:19:46.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-15T15:19:46.770 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:19:46.811 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 154/198 2024-04-15T15:19:46.842 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 155/198 2024-04-15T15:19:46.868 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-15T15:19:47.073 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 157/198 2024-04-15T15:19:47.188 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-15T15:19:47.234 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 159/198 2024-04-15T15:19:47.287 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-15T15:19:47.341 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-15T15:19:47.384 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-15T15:19:47.426 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 163/198 2024-04-15T15:19:47.449 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-15T15:19:47.479 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-15T15:19:47.530 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-15T15:19:47.591 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-15T15:19:47.629 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-15T15:19:48.212 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:19:48.261 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:19:48.281 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:19:49.159 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 170/198 2024-04-15T15:19:49.247 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-15T15:19:49.290 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-15T15:19:49.363 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-15T15:19:49.363 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:49.363 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-04-15T15:19:49.364 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-15T15:19:49.364 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-15T15:19:49.364 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-15T15:19:49.364 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-15T15:19:49.364 INFO:teuthology.orchestra.run.smithi012.stdout: acl-2.3.1-4.el9.x86_64 2024-04-15T15:19:49.364 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-15T15:19:49.365 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-15T15:19:49.365 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-15T15:19:49.365 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-15T15:19:49.365 INFO:teuthology.orchestra.run.smithi012.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-15T15:19:49.365 INFO:teuthology.orchestra.run.smithi012.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.365 INFO:teuthology.orchestra.run.smithi012.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.366 INFO:teuthology.orchestra.run.smithi012.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.366 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.366 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.366 INFO:teuthology.orchestra.run.smithi012.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.366 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.367 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-15T15:19:49.367 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-15T15:19:49.367 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.367 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.367 INFO:teuthology.orchestra.run.smithi012.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.367 INFO:teuthology.orchestra.run.smithi012.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-15T15:19:49.368 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-15T15:19:49.368 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-04-15T15:19:49.368 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-15T15:19:49.368 INFO:teuthology.orchestra.run.smithi012.stdout: automake-1.16.2-8.el9.noarch 2024-04-15T15:19:49.368 INFO:teuthology.orchestra.run.smithi012.stdout: bison-3.7.4-5.el9.x86_64 2024-04-15T15:19:49.368 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T15:19:49.369 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.369 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.369 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.369 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.369 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.369 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.370 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.370 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.370 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.370 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.371 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.371 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.371 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.371 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.371 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.372 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.372 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.372 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.372 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.372 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.372 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.373 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.373 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.373 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:49.373 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-15T15:19:49.373 INFO:teuthology.orchestra.run.smithi012.stdout: flex-2.6.4-9.el9.x86_64 2024-04-15T15:19:49.373 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T15:19:49.374 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T15:19:49.374 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T15:19:49.374 INFO:teuthology.orchestra.run.smithi012.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-15T15:19:49.374 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T15:19:49.374 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-15T15:19:49.375 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.6-16.el9.x86_64 2024-04-15T15:19:49.375 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-15T15:19:49.375 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T15:19:49.375 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T15:19:49.375 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.375 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.376 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.376 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T15:19:49.376 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T15:19:49.376 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T15:19:49.376 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T15:19:49.377 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T15:19:49.377 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.377 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.377 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T15:19:49.377 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.377 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-15T15:19:49.378 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T15:19:49.378 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-15T15:19:49.378 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-15T15:19:49.378 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T15:19:49.378 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2024-04-15T15:19:49.378 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-15T15:19:49.378 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-15T15:19:49.379 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T15:19:49.379 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T15:19:49.379 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T15:19:49.379 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-15T15:19:49.379 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T15:19:49.379 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-13.14-1.el9.x86_64 2024-04-15T15:19:49.379 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-15T15:19:49.380 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-15T15:19:49.380 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-15T15:19:49.380 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-15T15:19:49.380 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-15T15:19:49.380 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T15:19:49.380 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-15T15:19:49.380 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T15:19:49.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T15:19:49.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T15:19:49.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.381 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T15:19:49.382 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T15:19:49.382 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T15:19:49.382 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T15:19:49.382 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-15T15:19:49.382 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-15T15:19:49.382 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T15:19:49.382 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T15:19:49.383 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T15:19:49.383 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-15T15:19:49.383 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-15T15:19:49.383 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T15:19:49.383 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T15:19:49.383 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T15:19:49.383 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T15:19:49.384 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T15:19:49.384 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T15:19:49.384 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T15:19:49.384 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T15:19:49.384 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T15:19:49.384 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T15:19:49.384 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T15:19:49.384 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T15:19:49.385 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.386 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-15T15:19:49.387 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T15:19:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-15T15:19:49.389 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-15T15:19:49.389 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T15:19:49.389 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-15T15:19:49.390 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-15T15:19:49.390 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:19:49.390 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-04-15T15:19:49.401 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-15T15:19:49.426 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-15T15:19:49.451 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-15T15:19:49.478 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-15T15:19:49.478 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:19:49.574 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:19:49.604 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-15T15:19:49.604 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:19:49.689 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:19:49.716 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-15T15:19:49.756 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-15T15:19:49.782 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-15T15:19:49.806 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-15T15:19:49.825 DEBUG:teuthology.parallel:result is None 2024-04-15T15:19:49.850 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-15T15:19:49.887 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-15T15:19:49.920 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-15T15:19:49.945 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-15T15:19:49.970 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-15T15:19:49.995 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:19:50.021 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-15T15:19:50.027 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:19:50.027 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:19:50.216 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:19:50.216 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:19:50.281 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:19:50.313 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:19:50.340 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-15T15:19:50.357 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-15T15:19:50.391 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-15T15:19:50.459 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:19:55.856 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:19:55.857 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 1/198 2024-04-15T15:19:55.857 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2/198 2024-04-15T15:19:55.857 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 3/198 2024-04-15T15:19:55.857 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 4/198 2024-04-15T15:19:55.857 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 5/198 2024-04-15T15:19:55.857 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 6/198 2024-04-15T15:19:55.858 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 7/198 2024-04-15T15:19:55.858 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 8/198 2024-04-15T15:19:55.858 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 9/198 2024-04-15T15:19:55.858 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:19:55.858 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:19:55.858 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:19:55.859 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 13/198 2024-04-15T15:19:55.859 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 14/198 2024-04-15T15:19:55.859 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 15/198 2024-04-15T15:19:55.862 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:19:55.862 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 17/198 2024-04-15T15:19:55.862 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 18/198 2024-04-15T15:19:55.862 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 19/198 2024-04-15T15:19:55.862 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 20/198 2024-04-15T15:19:55.863 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 21/198 2024-04-15T15:19:55.863 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 22/198 2024-04-15T15:19:55.863 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 23/198 2024-04-15T15:19:55.863 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 24/198 2024-04-15T15:19:55.863 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 25/198 2024-04-15T15:19:55.863 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 26/198 2024-04-15T15:19:55.863 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 27/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 28/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 29/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 30/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 31/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 32/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 33/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 34/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 35/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 36/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 37/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 38/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 39/198 2024-04-15T15:19:55.864 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-15T15:19:55.865 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-15T15:19:55.866 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-15T15:19:55.867 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-15T15:19:55.868 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-15T15:19:55.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-15T15:19:55.870 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-15T15:19:55.871 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-15T15:19:55.872 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-15T15:19:55.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-15T15:19:55.874 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-15T15:19:55.875 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-15T15:19:55.876 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:19:55.877 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-15T15:19:55.877 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-15T15:19:55.877 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-15T15:19:58.873 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-15T15:19:58.873 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:58.873 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: acl-2.3.1-4.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-15T15:19:58.874 INFO:teuthology.orchestra.run.smithi055.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-15T15:19:58.875 INFO:teuthology.orchestra.run.smithi055.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: automake-1.16.2-8.el9.noarch 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: bison-3.7.4-5.el9.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.876 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.877 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: flex-2.6.4-9.el9.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-15T15:19:58.878 INFO:teuthology.orchestra.run.smithi055.stdout: jq-1.6-16.el9.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.879 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: lua-5.4.4-4.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T15:19:58.880 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-13.14-1.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.881 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-15T15:19:58.882 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T15:19:58.883 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T15:19:58.884 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T15:19:58.885 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:19:58.886 INFO:teuthology.orchestra.run.smithi055.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T15:19:58.887 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-15T15:19:58.887 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T15:19:58.887 INFO:teuthology.orchestra.run.smithi055.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-15T15:19:58.887 INFO:teuthology.orchestra.run.smithi055.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-15T15:19:58.887 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T15:19:58.887 INFO:teuthology.orchestra.run.smithi055.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-15T15:19:58.887 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:19:58.887 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-04-15T15:19:59.396 DEBUG:teuthology.parallel:result is None 2024-04-15T15:19:59.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:19:59.619 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T15:19:59.647 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:19:59.647 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:19:59.648 INFO:teuthology.task.install:The correct ceph version 19.0.0-2861.g9bf1c4ba is installed. 2024-04-15T15:19:59.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:19:59.902 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T15:19:59.929 INFO:teuthology.orchestra.run.smithi055.stdout:19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:19:59.929 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:19:59.929 INFO:teuthology.task.install:The correct ceph version 19.0.0-2861.g9bf1c4ba is installed. 2024-04-15T15:19:59.931 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-15T15:19:59.931 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:19:59.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T15:19:59.969 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:19:59.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T15:20:00.004 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-15T15:20:00.005 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:00.005 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T15:20:00.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T15:20:00.112 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:00.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T15:20:00.145 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T15:20:00.217 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-15T15:20:00.217 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:00.217 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T15:20:00.247 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T15:20:00.316 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:00.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T15:20:00.346 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T15:20:00.416 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-15T15:20:00.416 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:00.416 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T15:20:00.444 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T15:20:00.513 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:00.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T15:20:00.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T15:20:00.614 INFO:teuthology.run_tasks:Running task ceph... 2024-04-15T15:20:00.705 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-15T15:20:00.705 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2024-04-15T15:20:00.707 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /var/log/ceph 2024-04-15T15:20:00.738 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-15T15:20:00.738 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-15T15:20:00.776 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-15T15:20:00.808 INFO:tasks.ceph:Creating extra log directories... 2024-04-15T15:20:00.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-15T15:20:00.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-15T15:20:00.885 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-15T15:20:00.885 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-15T15:20:00.885 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-15T15:20:00.885 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:00.885 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-15T15:20:00.927 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-15T15:20:01.048 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-15T15:20:01.076 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:01.076 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-15T15:20:01.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-15T15:20:01.220 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-15T15:20:01.256 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-15T15:20:01.256 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-15T15:20:01.256 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144', 'branch': 'wip-lusov-quiesce-xlock', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce}', 'email': None, 'first_in_suite': False, 'job_id': '7657144', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-quiesce-xlock'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-quiesce-xlock', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5929, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-lusov-quiesce-xlock', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIE9jX1BIR5LrfW9u7p9ZFWARk8rZM4rxQ7Pj9xRDsPwJAkFNbl74+1mqPiy2sFSYQdawFpR9IBX0HvGSLEKDHk=', 'smithi055.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxUayo0doJBrH/sTt93Zv9BrOxzjdCJzWs0DlBISUHJ16DVvUvYhyj5tpb4iCkQc063omFSnO4e+x1yQTn2nKg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_quiesce']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-15_14:54:08', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-15T15:20:01.257 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-15T15:20:01.259 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-15T15:20:01.306 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2024-04-15T15:20:01.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/run/ceph 2024-04-15T15:20:01.376 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:01.377 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T15:20:01.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T15:20:01.398 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-04-15T15:20:01.453 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T15:20:01.453 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:20:01.453 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-15T15:20:01.454 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:20:01.454 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:20:01.454 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-15 15:19:41.332274575 +0000 2024-04-15T15:20:01.454 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-15 15:17:17.333343251 +0000 2024-04-15T15:20:01.454 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-15 15:17:17.333343251 +0000 2024-04-15T15:20:01.454 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-15 15:17:17.333343251 +0000 2024-04-15T15:20:01.454 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T15:20:01.520 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-15T15:20:01.520 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-15T15:20:01.520 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000189159 s, 2.7 MB/s 2024-04-15T15:20:01.521 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T15:20:01.577 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-04-15T15:20:01.632 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T15:20:01.633 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:20:01.633 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-15T15:20:01.633 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:20:01.633 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:20:01.633 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-15 15:19:41.332274575 +0000 2024-04-15T15:20:01.633 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-15 15:17:17.675340618 +0000 2024-04-15T15:20:01.633 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-15 15:17:17.675340618 +0000 2024-04-15T15:20:01.633 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-15 15:17:17.675340618 +0000 2024-04-15T15:20:01.633 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T15:20:01.699 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-15T15:20:01.700 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-15T15:20:01.700 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000277074 s, 1.8 MB/s 2024-04-15T15:20:01.701 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T15:20:01.757 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-15 15:19:41.332274575 +0000 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-15 15:17:18.017337986 +0000 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-15 15:17:18.017337986 +0000 2024-04-15T15:20:01.815 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-15 15:17:18.017337986 +0000 2024-04-15T15:20:01.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T15:20:01.881 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-15T15:20:01.881 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-15T15:20:01.882 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000181807 s, 2.8 MB/s 2024-04-15T15:20:01.883 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T15:20:01.939 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-04-15T15:20:01.994 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T15:20:01.995 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:20:01.995 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-15T15:20:01.995 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:20:01.995 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:20:01.995 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-15 15:19:41.333274567 +0000 2024-04-15T15:20:01.995 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-15 15:17:18.539333968 +0000 2024-04-15T15:20:01.995 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-15 15:17:18.539333968 +0000 2024-04-15T15:20:01.995 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-15 15:17:18.539333968 +0000 2024-04-15T15:20:01.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T15:20:02.062 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-15T15:20:02.062 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-15T15:20:02.062 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000196103 s, 2.6 MB/s 2024-04-15T15:20:02.063 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T15:20:02.119 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:02.119 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:02.119 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T15:20:02.137 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T15:20:02.138 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-04-15 15:19:51.064826435 +0000 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-04-15 15:17:18.402366020 +0000 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-04-15 15:17:18.402366020 +0000 2024-04-15T15:20:02.196 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-04-15 15:17:18.402366020 +0000 2024-04-15T15:20:02.197 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T15:20:02.267 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-04-15T15:20:02.267 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-04-15T15:20:02.267 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000322899 s, 1.6 MB/s 2024-04-15T15:20:02.268 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T15:20:02.327 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2024-04-15T15:20:02.385 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T15:20:02.385 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:20:02.385 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-15T15:20:02.385 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:20:02.385 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:20:02.386 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-04-15 15:19:51.065826414 +0000 2024-04-15T15:20:02.386 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-04-15 15:17:18.745359153 +0000 2024-04-15T15:20:02.386 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-04-15 15:17:18.745359153 +0000 2024-04-15T15:20:02.386 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-04-15 15:17:18.745359153 +0000 2024-04-15T15:20:02.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T15:20:02.454 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-04-15T15:20:02.455 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-04-15T15:20:02.455 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000255015 s, 2.0 MB/s 2024-04-15T15:20:02.456 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T15:20:02.514 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2024-04-15T15:20:02.569 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T15:20:02.570 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:20:02.570 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-15T15:20:02.570 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:20:02.570 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:20:02.570 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-04-15 15:19:51.065826414 +0000 2024-04-15T15:20:02.570 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-04-15 15:17:19.104351966 +0000 2024-04-15T15:20:02.570 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-04-15 15:17:19.104351966 +0000 2024-04-15T15:20:02.570 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-04-15 15:17:19.104351966 +0000 2024-04-15T15:20:02.571 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T15:20:02.638 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-04-15T15:20:02.639 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-04-15T15:20:02.639 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000286088 s, 1.8 MB/s 2024-04-15T15:20:02.640 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T15:20:02.697 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2024-04-15T15:20:02.753 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T15:20:02.753 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:20:02.753 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-15T15:20:02.753 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:20:02.753 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:20:02.753 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-04-15 15:19:51.066826394 +0000 2024-04-15T15:20:02.754 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-04-15 15:17:19.453344979 +0000 2024-04-15T15:20:02.754 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-04-15 15:17:19.453344979 +0000 2024-04-15T15:20:02.754 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-04-15 15:17:19.453344979 +0000 2024-04-15T15:20:02.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T15:20:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-04-15T15:20:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-04-15T15:20:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000274405 s, 1.9 MB/s 2024-04-15T15:20:02.822 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T15:20:02.879 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:02.879 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-15T15:20:02.879 INFO:tasks.ceph:Generating config... 2024-04-15T15:20:02.881 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-15T15:20:02.881 INFO:tasks.ceph:[client] debug client = 20 2024-04-15T15:20:02.881 INFO:tasks.ceph:[client] debug ms = 1 2024-04-15T15:20:02.881 INFO:tasks.ceph:[client] pid file = 2024-04-15T15:20:02.881 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-15T15:20:02.881 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-15T15:20:02.881 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-15T15:20:02.881 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-15T15:20:02.881 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-15T15:20:02.881 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-15T15:20:02.881 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-15T15:20:02.881 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-15T15:20:02.881 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-15T15:20:02.882 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-15T15:20:02.882 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-15T15:20:02.882 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-15T15:20:02.882 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-15T15:20:02.883 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-15T15:20:02.883 INFO:tasks.ceph:Setting up mon.a... 2024-04-15T15:20:02.884 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-15T15:20:02.941 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.keyring 2024-04-15T15:20:02.946 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-15T15:20:02.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:20:03.025 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.12'), ('mon.b', '172.21.15.55'), ('mon.c', '[v2:172.21.15.55:3301,v1:172.21.15.55:6790]')] 2024-04-15T15:20:03.026 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.12,172.21.15.55,[v2:172.21.15.55:3301,v1:172.21.15.55:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-15T15:20:03.027 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:03.027 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-15T15:20:03.082 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.12 --add b 172.21.15.55 --addv c '[v2:172.21.15.55:3301,v1:172.21.15.55:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:03.170 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:03.170 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid 928fa63b-2322-46eb-a6b8-ef2b880cd7c9 2024-04-15T15:20:03.170 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = quincy 2024-04-15T15:20:03.170 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2024-04-15T15:20:03.170 INFO:teuthology.orchestra.run.smithi012.stdout:fsid 928fa63b-2322-46eb-a6b8-ef2b880cd7c9 2024-04-15T15:20:03.171 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2024-04-15T15:20:03.169614+0000 2024-04-15T15:20:03.171 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-04-15T15:20:03.169614+0000 2024-04-15T15:20:03.171 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 17 (quincy) 2024-04-15T15:20:03.171 INFO:teuthology.orchestra.run.smithi012.stdout:election_strategy: 1 2024-04-15T15:20:03.171 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-15T15:20:03.171 INFO:teuthology.orchestra.run.smithi012.stdout:1: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.b 2024-04-15T15:20:03.171 INFO:teuthology.orchestra.run.smithi012.stdout:2: [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] mon.c 2024-04-15T15:20:03.171 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-15T15:20:03.172 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-15T15:20:03.187 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 928fa63b-2322-46eb-a6b8-ef2b880cd7c9... 2024-04-15T15:20:03.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-15T15:20:03.230 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-15T15:20:03.282 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-04-15T15:20:03.282 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2024-04-15T15:20:03.282 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-15T15:20:03.282 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2024-04-15T15:20:03.282 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.282 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2024-04-15T15:20:03.282 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.282 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.283 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-15T15:20:03.284 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = "172.21.15.12,172.21.15.55,[v2:172.21.15.55:3301,v1:172.21.15.55:6790]" 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 928fa63b-2322-46eb-a6b8-ef2b880cd7c9 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2024-04-15T15:20:03.285 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug trim objects = true 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.286 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore block size = 96636764160 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore fsck on mount = True 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluefs = 20 2024-04-15T15:20:03.287 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluestore = 20 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: debug rocksdb = 10 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd backfillfull_ratio = 0.85 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd full ratio = 0.9 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd nearfull ratio = 0.8 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: osd failsafe full ratio = 0.95 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = bluestore 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: osd op complaint time = 180 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-04-15T15:20:03.288 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: debug client = 20 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:20:03.289 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2024-04-15T15:20:03.290 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: mon down mkfs grace = 300 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: mon op complaint time = 120 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: client mount timeout = 600 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: debug client = 20 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-04-15T15:20:03.291 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = "" 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout: rados mon op timeout = 900 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout: rados osd op timeout = 900 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.b] 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.c] 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a] 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c] 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b] 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d] 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout:[mds] 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout: debug mds = 20 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout: debug mds balancer = 20 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout: mds debug frag = True 2024-04-15T15:20:03.292 INFO:teuthology.orchestra.run.smithi012.stdout: mds debug scatterstat = True 2024-04-15T15:20:03.293 INFO:teuthology.orchestra.run.smithi012.stdout: mds op complaint time = 180 2024-04-15T15:20:03.293 INFO:teuthology.orchestra.run.smithi012.stdout: mds verify scatter = True 2024-04-15T15:20:03.293 INFO:teuthology.orchestra.run.smithi012.stdout: osd op complaint time = 180 2024-04-15T15:20:03.293 INFO:teuthology.orchestra.run.smithi012.stdout: rados mon op timeout = 900 2024-04-15T15:20:03.293 INFO:teuthology.orchestra.run.smithi012.stdout: rados osd op timeout = 900 2024-04-15T15:20:03.293 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2024-04-15T15:20:03.293 INFO:teuthology.orchestra.run.smithi055.stdout: chdir = "" 2024-04-15T15:20:03.293 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: auth supported = cephx 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: filestore xattr use omap = true 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: mon clock drift allowed = 1.000 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: osd crush chooseleaf type = 0 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: auth debug = true 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on old message = true 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on bug = true 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-15T15:20:03.294 INFO:teuthology.orchestra.run.smithi055.stdout: mon pg warn max object skew = 0 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: # disable pg_autoscaler by default for new pools 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default size = 2 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow primary affinity = true 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow pg remap = true 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on legacy crush tunables = false 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on crush straw calc version zero = false 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on no sortbitwise = false 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on osd down out interval zero = false 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on too few osds = false 2024-04-15T15:20:03.295 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: mon_allow_pool_size_one = true 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: osd default data pool replay window = 5 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: mon allow pool delete = true 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: mon cluster log file level = debug 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: debug asserts on shutdown = true 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: mon health detail to clog = false 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: mon host = "172.21.15.12,172.21.15.55,[v2:172.21.15.55:3301,v1:172.21.15.55:6790]" 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = 928fa63b-2322-46eb-a6b8-ef2b880cd7c9 2024-04-15T15:20:03.296 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout:[osd] 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd journal size = 100 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub load threshold = 5.0 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub max interval = 600 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock profile = high_recovery_ops 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd recover clone overlap = true 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd recovery max chunk = 1048576 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug shutdown = true 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug op order = true 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify stray on activate = true 2024-04-15T15:20:03.297 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug trim objects = true 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: osd open classes on start = true 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug pg log writeout = true 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: osd deep scrub update digest min age = 30 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: osd map max advance = 10 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: journal zero on create = true 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: filestore ondisk finisher threads = 3 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: filestore apply finisher threads = 3 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.298 INFO:teuthology.orchestra.run.smithi055.stdout: bdev debug aio = true 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug misdirected ops = true 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore block size = 96636764160 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore fsck on mount = True 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluefs = 20 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluestore = 20 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: debug osd = 20 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: debug rocksdb = 10 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd backfillfull_ratio = 0.85 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd full ratio = 0.9 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd nearfull ratio = 0.8 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: osd failsafe full ratio = 0.95 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: osd objectstore = bluestore 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: osd op complaint time = 180 2024-04-15T15:20:03.299 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr] 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: debug mgr = 20 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: mgr/telemetry/nag = false 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: debug client = 20 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout:[mon] 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2024-04-15T15:20:03.300 INFO:teuthology.orchestra.run.smithi055.stdout: debug paxos = 20 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon data avail warn = 5 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon mgr mkfs grace = 240 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd reporter subtree level = osd 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd prime pg temp = true 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: auth mon ticket ttl = 660 # 11m 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: auth service ticket ttl = 240 # 4m 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: # don't complain about insecure global_id in the test suite 2024-04-15T15:20:03.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: # 1m isn't quite enough 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: mon_down_mkfs_grace = 2m 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_filestore_osds = false 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: mon down mkfs grace = 300 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: mon op complaint time = 120 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout:[client] 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: rgw cache enabled = true 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable ops log = true 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable usage log = true 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-15T15:20:03.302 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout: client mount timeout = 600 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout: debug client = 20 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = "" 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout: rados mon op timeout = 900 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout: rados osd op timeout = 900 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.a] 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.b] 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.c] 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.a] 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.c] 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.b] 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.d] 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout:[mds] 2024-04-15T15:20:03.303 INFO:teuthology.orchestra.run.smithi055.stdout: debug mds = 20 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: debug mds balancer = 20 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: mds debug frag = True 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: mds debug scatterstat = True 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: mds op complaint time = 180 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: mds verify scatter = True 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: osd op complaint time = 180 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: rados mon op timeout = 900 2024-04-15T15:20:03.304 INFO:teuthology.orchestra.run.smithi055.stdout: rados osd op timeout = 900 2024-04-15T15:20:03.306 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-15T15:20:03.306 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-15T15:20:03.404 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-15T15:20:03.405 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:03.405 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-15T15:20:03.420 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:03.420 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-15T15:20:03.475 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:20:03.476 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:03.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-15T15:20:03.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:20:03.557 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:03.557 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:03.612 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:20:03.612 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:03.612 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-15T15:20:03.612 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:20:03.660 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:03.660 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:03.718 INFO:tasks.ceph:Setting up mon nodes... 2024-04-15T15:20:03.718 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-15T15:20:03.718 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-15T15:20:03.775 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-15T15:20:03.777 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-15T15:20:03.851 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-15T15:20:03.854 INFO:tasks.ceph:Setting up mds nodes... 2024-04-15T15:20:03.854 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-15T15:20:03.911 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-15T15:20:03.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-15T15:20:03.941 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-15T15:20:04.038 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-15T15:20:04.041 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-15T15:20:04.070 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-15T15:20:04.133 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-15T15:20:04.136 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-15T15:20:04.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-15T15:20:04.264 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-15T15:20:04.267 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-15T15:20:04.294 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-15T15:20:04.294 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T15:20:04.340 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-15T15:20:04.356 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-15T15:20:04.399 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-15T15:20:04.415 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-15T15:20:04.459 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-15T15:20:04.475 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-15T15:20:04.559 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-15T15:20:04.574 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-15T15:20:04.575 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-15T15:20:04.575 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-15T15:20:04.603 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:04.603 INFO:tasks.ceph:role: osd.0 2024-04-15T15:20:04.603 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:20:04.603 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:20:04.678 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:20:04.679 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:20:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-04-15T15:20:05.192 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-04-15T15:20:05.192 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-15T15:20:05.329 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-15T15:20:05.366 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-15T15:20:05.436 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:05.436 INFO:tasks.ceph:role: osd.1 2024-04-15T15:20:05.436 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:20:05.436 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-15T15:20:05.512 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:20:05.512 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:20:05.512 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:20:05.512 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:20:05.513 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:20:05.513 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:20:05.513 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:20:05.513 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:20:05.513 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:20:05.513 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:20:05.967 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-04-15T15:20:05.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-04-15T15:20:05.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-15T15:20:06.074 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-15T15:20:06.109 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-15T15:20:06.178 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:06.179 INFO:tasks.ceph:role: osd.2 2024-04-15T15:20:06.179 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:20:06.179 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-15T15:20:06.256 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:20:06.256 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:20:06.256 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:20:06.256 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:20:06.256 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:20:06.256 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:20:06.256 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:20:06.256 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:20:06.257 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:20:06.257 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:20:06.758 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-04-15T15:20:06.759 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-04-15T15:20:06.760 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-15T15:20:06.906 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-15T15:20:06.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-15T15:20:07.015 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:07.015 INFO:tasks.ceph:role: osd.3 2024-04-15T15:20:07.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:20:07.015 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-15T15:20:07.092 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:20:07.092 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:20:07.092 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:20:07.093 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:20:07.093 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:20:07.093 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:20:07.093 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:20:07.093 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:20:07.093 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:20:07.093 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:20:07.581 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-04-15T15:20:07.584 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-04-15T15:20:07.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-15T15:20:07.710 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-15T15:20:07.748 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:07.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:07.848+0000 7f96de511640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-15T15:20:07.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:07.848+0000 7f96de511640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-15T15:20:07.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:07.848+0000 7f96de511640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:20:07.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:07.848+0000 7f96de511640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:20:07.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:07.848+0000 7f96de511640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:20:07.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:07.848+0000 7f96de511640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-15T15:20:10.704 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-15T15:20:10.737 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:10.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:10.830+0000 7f466720d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-15T15:20:10.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:10.830+0000 7f466720d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-15T15:20:10.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:10.830+0000 7f466720d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:20:10.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:10.830+0000 7f466720d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:20:10.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:10.830+0000 7f466720d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:20:10.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:10.831+0000 7f466720d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-15T15:20:13.693 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-15T15:20:13.726 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:13.819+0000 7f28ac324640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-15T15:20:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:13.819+0000 7f28ac324640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-15T15:20:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:13.819+0000 7f28ac324640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:20:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:13.819+0000 7f28ac324640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:20:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:13.819+0000 7f28ac324640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:20:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:13.820+0000 7f28ac324640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-15T15:20:16.681 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-15T15:20:16.714 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:16.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:16.805+0000 7f596cea6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-15T15:20:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:16.805+0000 7f596cea6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-15T15:20:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:16.805+0000 7f596cea6640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:20:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:16.805+0000 7f596cea6640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:20:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:16.805+0000 7f596cea6640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:20:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:16.806+0000 7f596cea6640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-15T15:20:19.653 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-15T15:20:19.702 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-15T15:20:19.735 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:19.736 INFO:tasks.ceph:role: osd.4 2024-04-15T15:20:19.736 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:20:19.736 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-15T15:20:19.835 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:20:19.835 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:20:19.835 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:20:19.835 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:20:19.835 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:20:19.836 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:20:19.836 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:20:19.836 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:20:19.836 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:20:19.836 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:20:21.154 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-04-15T15:20:21.156 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-04-15T15:20:21.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-15T15:20:21.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-15T15:20:21.275 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-15T15:20:21.346 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:21.346 INFO:tasks.ceph:role: osd.5 2024-04-15T15:20:21.346 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:20:21.346 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-15T15:20:21.445 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:20:21.445 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:20:21.445 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:20:21.445 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:20:21.446 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:20:21.446 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:20:21.446 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:20:21.446 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:20:21.446 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:20:21.446 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:20:22.762 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-04-15T15:20:22.764 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-04-15T15:20:22.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-15T15:20:22.800 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-15T15:20:22.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-15T15:20:22.942 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:22.942 INFO:tasks.ceph:role: osd.6 2024-04-15T15:20:22.942 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:20:22.942 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-15T15:20:23.037 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:20:23.038 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:20:24.376 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-04-15T15:20:24.378 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-04-15T15:20:24.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-15T15:20:24.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-15T15:20:24.494 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-15T15:20:24.565 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:20:24.565 INFO:tasks.ceph:role: osd.7 2024-04-15T15:20:24.565 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:20:24.565 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-15T15:20:24.662 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:20:24.663 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:20:25.981 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-04-15T15:20:25.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-04-15T15:20:25.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-15T15:20:26.023 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-15T15:20:26.099 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:26.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:26.199+0000 7f2b2881c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-15T15:20:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:26.200+0000 7f2b2881c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-15T15:20:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:26.200+0000 7f2b2881c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:20:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:26.200+0000 7f2b2881c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:20:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:26.200+0000 7f2b2881c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:20:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:26.201+0000 7f2b2881c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-15T15:20:29.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-15T15:20:29.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:29.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:29.206+0000 7f9ba2813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-15T15:20:29.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:29.207+0000 7f9ba2813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-15T15:20:29.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:29.207+0000 7f9ba2813640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:20:29.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:29.207+0000 7f9ba2813640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:20:29.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:29.207+0000 7f9ba2813640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:20:29.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:29.207+0000 7f9ba2813640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-15T15:20:31.258 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:20:31.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:20:32.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-15T15:20:32.116 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:32.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:32.206+0000 7fcf3a39a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-15T15:20:32.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:32.206+0000 7fcf3a39a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-15T15:20:32.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:32.206+0000 7fcf3a39a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:20:32.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:32.206+0000 7fcf3a39a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:20:32.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:32.206+0000 7fcf3a39a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:20:32.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:32.207+0000 7fcf3a39a640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-15T15:20:35.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-15T15:20:35.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:35.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:35.202+0000 7f5123099640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-15T15:20:35.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:35.202+0000 7f5123099640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-15T15:20:35.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:35.202+0000 7f5123099640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:20:35.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:35.202+0000 7f5123099640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:20:35.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:35.202+0000 7f5123099640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:20:35.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:35.203+0000 7f5123099640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-15T15:20:38.072 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-15T15:20:38.105 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-15T15:20:38.105 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:38.105 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-15T15:20:38.138 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:38.138 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-15T15:20:38.210 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:38.210 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-15T15:20:38.281 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:38.281 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-15T15:20:38.351 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:38.351 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-15T15:20:38.421 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:38.422 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-15T15:20:38.492 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:38.492 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-15T15:20:38.561 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:38.562 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-15T15:20:38.594 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:38.594 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-15T15:20:38.665 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:38.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-15T15:20:38.736 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:38.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-15T15:20:38.808 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:38.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-15T15:20:38.880 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:38.880 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-15T15:20:38.951 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:38.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-15T15:20:39.021 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:39.022 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-15T15:20:39.040 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:39.040 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-15T15:20:39.099 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:39.099 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-15T15:20:39.115 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:39.116 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-15T15:20:39.171 INFO:tasks.ceph:Adding keys to all mons... 2024-04-15T15:20:39.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-15T15:20:39.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-15T15:20:39.214 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.y] 2024-04-15T15:20:39.214 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCjRR1myBgtLhAA7md6R7R4ntQzqtiyaAh3iw== 2024-04-15T15:20:39.214 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a] 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCjRR1mfGBLNhAAzcTNRabmma9rGvlCj4vgFA== 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c] 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCkRR1m1fBGAhAAvTFwjNN+5kD3j429N9eQ/A== 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCnRR1mDOuUMhAA+hU2FOSOsTeRWQXuSouJSw== 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCqRR1m+puDMRAATMzipAvsvuQZXd04A5WtgA== 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCtRR1m9qnfMBAA4rNHZSIpzmSqQ2RpWEQYdg== 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.3] 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCwRR1m4r0ZMBAAOHHdTfsxkvI2I8i9HNbO1g== 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2024-04-15T15:20:39.215 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCjRR1mDPm2MhAAK0Z9rdK9KYolx2DI6EBTvw== 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b] 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCkRR1mncnxBxAAWy5Phi+MopzEoKYt7WlTXA== 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d] 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCkRR1mUtW6DxAAZKDeHi4Kmfhulx8nbIYepg== 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.4] 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQC6RR1mwNH1CxAAkQq3vwEtpL7MdIpq+368QQ== 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.5] 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQC9RR1mezNhDBAA4Py3odTAM0kR7yfal3jPCA== 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.6] 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQDARR1m5GtZDBAAahBaCoZPZHsetB07+L/BEQ== 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.7] 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQDDRR1m5hIsDBAAiUoogjGRfr6KeRjxU/9YTA== 2024-04-15T15:20:39.216 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-04-15T15:20:39.217 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCkRR1muiw8FBAAiEh+BBQRtVSuiPlO3ZezfA== 2024-04-15T15:20:39.217 INFO:teuthology.orchestra.run.smithi012.stdout:[client.2] 2024-04-15T15:20:39.217 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCkRR1mWDLDFxAAGuwFUqNroSjT5XBZOXyONQ== 2024-04-15T15:20:39.217 INFO:teuthology.orchestra.run.smithi012.stdout:[client.1] 2024-04-15T15:20:39.217 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCkRR1mQ4RTGxAAiJY6pHGO9TIYvUoQyoqwpw== 2024-04-15T15:20:39.217 INFO:teuthology.orchestra.run.smithi012.stdout:[client.3] 2024-04-15T15:20:39.217 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCkRR1mZuxIIRAAOKEZ5qaFn2en0OyexnZX9A== 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.y] 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCjRR1myBgtLhAA7md6R7R4ntQzqtiyaAh3iw== 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.a] 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCjRR1mfGBLNhAAzcTNRabmma9rGvlCj4vgFA== 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.c] 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCkRR1m1fBGAhAAvTFwjNN+5kD3j429N9eQ/A== 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.0] 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCnRR1mDOuUMhAA+hU2FOSOsTeRWQXuSouJSw== 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.1] 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCqRR1m+puDMRAATMzipAvsvuQZXd04A5WtgA== 2024-04-15T15:20:39.239 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.2] 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCtRR1m9qnfMBAA4rNHZSIpzmSqQ2RpWEQYdg== 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.3] 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCwRR1m4r0ZMBAAOHHdTfsxkvI2I8i9HNbO1g== 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.x] 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCjRR1mDPm2MhAAK0Z9rdK9KYolx2DI6EBTvw== 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.b] 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCkRR1mncnxBxAAWy5Phi+MopzEoKYt7WlTXA== 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.d] 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCkRR1mUtW6DxAAZKDeHi4Kmfhulx8nbIYepg== 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.4] 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQC6RR1mwNH1CxAAkQq3vwEtpL7MdIpq+368QQ== 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.5] 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQC9RR1mezNhDBAA4Py3odTAM0kR7yfal3jPCA== 2024-04-15T15:20:39.240 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.6] 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDARR1m5GtZDBAAahBaCoZPZHsetB07+L/BEQ== 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.7] 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQDDRR1m5hIsDBAAiUoogjGRfr6KeRjxU/9YTA== 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCkRR1muiw8FBAAiEh+BBQRtVSuiPlO3ZezfA== 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout:[client.2] 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCkRR1mWDLDFxAAGuwFUqNroSjT5XBZOXyONQ== 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout:[client.1] 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCkRR1mQ4RTGxAAiJY6pHGO9TIYvUoQyoqwpw== 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout:[client.3] 2024-04-15T15:20:39.241 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCkRR1mZuxIIRAAOKEZ5qaFn2en0OyexnZX9A== 2024-04-15T15:20:39.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:20:39.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:20:39.338 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:20:39.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:20:39.398 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:20:39.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:20:39.457 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.459 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.512 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.623 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.679 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:20:39.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:20:39.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:20:39.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:20:39.794 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:20:39.797 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:20:39.854 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.910 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.912 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.965 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:39.968 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:40.021 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:40.024 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:20:40.077 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:20:40.080 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:20:40.136 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:20:40.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:20:40.195 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:20:40.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:20:40.254 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:20:40.257 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:20:40.314 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-15T15:20:40.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-15T15:20:40.342 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:20:40.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-15T15:20:40.470 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-15T15:20:40.498 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:20:40.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-15T15:20:40.644 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-15T15:20:40.712 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:20:40.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-15T15:20:40.853 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:40.855 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:20:40.909 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-15T15:20:40.909 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-15T15:20:40.909 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-15T15:20:40.911 INFO:tasks.ceph.mon.a:Started 2024-04-15T15:20:40.912 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-15T15:20:40.912 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-15T15:20:40.951 INFO:tasks.ceph.mon.b:Started 2024-04-15T15:20:40.951 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-15T15:20:40.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-15T15:20:40.953 INFO:tasks.ceph.mon.c:Started 2024-04-15T15:20:40.953 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-15T15:20:40.953 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-15T15:20:40.953 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-15T15:20:40.955 INFO:tasks.ceph.mgr.y:Started 2024-04-15T15:20:40.956 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-15T15:20:40.956 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-15T15:20:40.958 INFO:tasks.ceph.mgr.x:Started 2024-04-15T15:20:40.958 INFO:tasks.ceph:Setting crush tunables to default 2024-04-15T15:20:40.958 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd crush tunables default 2024-04-15T15:20:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.063+0000 7f3312ea5640 1 Processor -- start 2024-04-15T15:20:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.063+0000 7f3312ea5640 1 -- start start 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3312ea5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3312ea5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3312ea5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f330c15ba70 con 0x7f330c15b690 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3312ea5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f330c0d0d50 con 0x7f330c0d08a0 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3312ea5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f330c0ce5b0 con 0x7f330c0d1a10 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3311442640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f330bfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3310c41640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f330c15b690 0x7f330c0d1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:40378/0 (socket says 172.21.15.12:40378) 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 learned_addr learned my addr 172.21.15.12:0/2645766889 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:32938/0 (socket says 172.21.15.12:32938) 2024-04-15T15:20:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-15T15:20:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:20:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T15:20:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.064+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T15:20:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:20:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T15:20:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-15T15:20:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.265+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T15:20:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-15T15:20:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-15T15:20:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:20:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:41.666+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-15T15:20:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:20:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-15T15:20:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-15T15:20:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:42.467+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-15T15:20:44.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f330a7fc640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 msgr2=0x7f330c0b6ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:20:44.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f330a7fc640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f330c0d1a10 0x7f330c0b6ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:44.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f330a7fc640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 msgr2=0x7f330c0b1660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:20:44.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f330a7fc640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.55:3300/0 conn(0x7f330c0d08a0 0x7f330c0b1660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:44.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f330a7fc640 1 -- 172.21.15.12:0/2645766889 >> v1:172.21.15.55:6789/0 conn(0x7f330c15b690 legacy=0x7f330c0d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f330a7fc640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f330a7fc640 1 -- 172.21.15.12:0/2645766889 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f330c0ce890 con 0x7f32f8006190 2024-04-15T15:20:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f330a7fc640 1 -- 172.21.15.12:0/2645766889 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f330c0d0d50 con 0x7f32f80036e0 2024-04-15T15:20:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:20:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.065+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T15:20:44.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.266+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:44.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.266+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:20:44.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.266+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:44.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.266+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:44.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.266+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T15:20:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.667+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.667+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:20:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.667+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.667+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:44.667+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-15T15:20:45.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:45.468+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:45.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:45.468+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:20:45.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:45.468+0000 7f3310c41640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:20:45.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:45.468+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:20:45.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:45.468+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-15T15:20:46.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.250+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2520929367 0 0) 0x7f330c0ce890 con 0x7f32f8006190 2024-04-15T15:20:46.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.250+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32f4003520 con 0x7f32f8006190 2024-04-15T15:20:46.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3679838064 0 0) 0x7f32f4003520 con 0x7f32f8006190 2024-04-15T15:20:46.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 msgr2=0x7f32f8005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:20:46.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 --2- 172.21.15.12:0/2645766889 >> v2:172.21.15.12:3300/0 conn(0x7f32f80036e0 0x7f32f8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f800af70 con 0x7f32f8006190 2024-04-15T15:20:46.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f330000f790 con 0x7f32f8006190 2024-04-15T15:20:46.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 >> v1:172.21.15.12:6789/0 conn(0x7f32f8006190 legacy=0x7f32f8008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f3311442640 1 -- 172.21.15.12:0/2645766889 reap_dead start 2024-04-15T15:20:46.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f40039b0 0x7f32f4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f4004800 0x7f32f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40052e0 0x7f32f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f330c0d0d50 con 0x7f32f4004800 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f32f400c230 con 0x7f32f40039b0 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.251+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f32f400c430 con 0x7f32f40052e0 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.252+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f4004800 0x7f32f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.252+0000 7f3311442640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40052e0 0x7f32f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.252+0000 7f3310c41640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f40039b0 0x7f32f4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.252+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40052e0 msgr2=0x7f32f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.252+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40052e0 0x7f32f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.252+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f40039b0 msgr2=0x7f32f4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.252+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f40039b0 0x7f32f4003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.252+0000 7f330bfff640 1 -- 172.21.15.12:0/2645766889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f4003520 con 0x7f32f4004800 2024-04-15T15:20:46.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.253+0000 7f330bfff640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f4004800 0x7f32f4004c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3300006fd0 tx=0x7f32f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d96eda2293bed03 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.254+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300037070 con 0x7f32f4004800 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.254+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3300033630 con 0x7f32f4004800 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.254+0000 7f330b7fe640 1 -- 172.21.15.12:0/2645766889 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300012400 con 0x7f32f4004800 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.255+0000 7f3312ea5640 1 -- 172.21.15.12:0/2645766889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f4004800 msgr2=0x7f32f4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.255+0000 7f3312ea5640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f4004800 0x7f32f4004c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3300006fd0 tx=0x7f32f4005830 comp rx=0 tx=0).stop 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.255+0000 7f3312ea5640 1 -- 172.21.15.12:0/2645766889 shutdown_connections 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.255+0000 7f3312ea5640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40052e0 0x7f32f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.255+0000 7f3312ea5640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f4004800 0x7f32f4004c80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.255+0000 7f3312ea5640 1 --2- 172.21.15.12:0/2645766889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f40039b0 0x7f32f4003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.255+0000 7f3312ea5640 1 -- 172.21.15.12:0/2645766889 >> 172.21.15.12:0/2645766889 conn(0x7f330c09a9d0 msgr2=0x7f330c0b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:46.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.256+0000 7f3312ea5640 1 -- 172.21.15.12:0/2645766889 shutdown_connections 2024-04-15T15:20:46.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.256+0000 7f3312ea5640 1 -- 172.21.15.12:0/2645766889 wait complete. 2024-04-15T15:20:46.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.257+0000 7f3312ea5640 1 Processor -- start 2024-04-15T15:20:46.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.257+0000 7f3312ea5640 1 -- start start 2024-04-15T15:20:46.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.258+0000 7f3312ea5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40039b0 0x7f330c143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.258+0000 7f3312ea5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f4004800 0x7f330c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.258+0000 7f3312ea5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f40052e0 0x7f330c151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.258+0000 7f3312ea5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f330c0b6ff0 con 0x7f32f40052e0 2024-04-15T15:20:46.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.258+0000 7f3312ea5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f330c0d1010 con 0x7f32f4004800 2024-04-15T15:20:46.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3312ea5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f330c0b7170 con 0x7f32f40039b0 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3310c41640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40039b0 0x7f330c143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3311442640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f40052e0 0x7f330c151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f330bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f4004800 0x7f330c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f330bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f4004800 0x7f330c150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:40370/0 (socket says 172.21.15.12:40370) 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3310c41640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40039b0 0x7f330c143c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:51504/0 (socket says 172.21.15.12:51504) 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f330bfff640 1 -- 172.21.15.12:0/653129607 learned_addr learned my addr 172.21.15.12:0/653129607 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3311442640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f40052e0 0x7f330c151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60014/0 (socket says 172.21.15.12:60014) 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3311442640 1 -- 172.21.15.12:0/653129607 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40039b0 msgr2=0x7f330c143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3311442640 1 --2- 172.21.15.12:0/653129607 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40039b0 0x7f330c143c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3311442640 1 -- 172.21.15.12:0/653129607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f4004800 msgr2=0x7f330c150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3311442640 1 --2- 172.21.15.12:0/653129607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f4004800 0x7f330c150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3311442640 1 -- 172.21.15.12:0/653129607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c0b47c0 con 0x7f32f40052e0 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f3311442640 1 --2- 172.21.15.12:0/653129607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f40052e0 0x7f330c151360 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f330c072710 tx=0x7f3304004910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:46.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.259+0000 7f33097fa640 1 -- 172.21.15.12:0/653129607 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330401c030 con 0x7f32f40052e0 2024-04-15T15:20:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.260+0000 7f33097fa640 1 -- 172.21.15.12:0/653129607 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3304015b30 con 0x7f32f40052e0 2024-04-15T15:20:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.260+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330c14f120 con 0x7f32f40052e0 2024-04-15T15:20:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.260+0000 7f33097fa640 1 -- 172.21.15.12:0/653129607 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3304021410 con 0x7f32f40052e0 2024-04-15T15:20:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.260+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330c14f6e0 con 0x7f32f40052e0 2024-04-15T15:20:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.260+0000 7f33097fa640 1 -- 172.21.15.12:0/653129607 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3304005d10 con 0x7f32f40052e0 2024-04-15T15:20:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.260+0000 7f33097fa640 1 -- 172.21.15.12:0/653129607 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f33040203c0 con 0x7f32f40052e0 2024-04-15T15:20:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.261+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f330c0690f0 con 0x7f32f40052e0 2024-04-15T15:20:46.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.265+0000 7f33097fa640 1 -- 172.21.15.12:0/653129607 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3304026070 con 0x7f32f40052e0 2024-04-15T15:20:46.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.333+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f330c0b69e0 con 0x7f32f40052e0 2024-04-15T15:20:46.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.335+0000 7f33097fa640 1 -- 172.21.15.12:0/653129607 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f3304010050 con 0x7f32f40052e0 2024-04-15T15:20:46.338 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.336+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f40052e0 msgr2=0x7f330c151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.336+0000 7f3312ea5640 1 --2- 172.21.15.12:0/653129607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f40052e0 0x7f330c151360 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f330c072710 tx=0x7f3304004910 comp rx=0 tx=0).stop 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.336+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 shutdown_connections 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.336+0000 7f3312ea5640 1 --2- 172.21.15.12:0/653129607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32f40052e0 0x7f330c151360 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.336+0000 7f3312ea5640 1 --2- 172.21.15.12:0/653129607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32f4004800 0x7f330c150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.336+0000 7f3312ea5640 1 --2- 172.21.15.12:0/653129607 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f32f40039b0 0x7f330c143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.336+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 >> 172.21.15.12:0/653129607 conn(0x7f330c09a9d0 msgr2=0x7f330c155700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.337+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 shutdown_connections 2024-04-15T15:20:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:46.337+0000 7f3312ea5640 1 -- 172.21.15.12:0/653129607 wait complete. 2024-04-15T15:20:46.350 INFO:tasks.ceph:check_enable_crimson: False 2024-04-15T15:20:46.350 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-15T15:20:46.351 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:46.351 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-15T15:20:46.380 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:46.381 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-15T15:20:46.449 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:46.449 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-15T15:20:46.517 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:20:46.517 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-15T15:20:46.585 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:46.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-15T15:20:46.614 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:46.615 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-15T15:20:46.683 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:46.683 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-15T15:20:46.738 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:46.737+0000 7fc6f7041040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T15:20:46.751 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:20:46.751 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-15T15:20:46.756 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:46.753+0000 7fb2824da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T15:20:46.796 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:46.794+0000 7fc6f7041040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T15:20:46.812 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:46.810+0000 7fb2824da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T15:20:46.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd new 9af3971f-5436-411b-bd33-a0ec8913f792 0 2024-04-15T15:20:46.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.961+0000 7f84e0274640 1 Processor -- start 2024-04-15T15:20:46.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.961+0000 7f84e0274640 1 -- start start 2024-04-15T15:20:46.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.962+0000 7f84e0274640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f84d8071180 0x7f84d809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.962+0000 7f84e0274640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d80716f0 con 0x7f84d815ff10 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.962+0000 7f84e0274640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d80d0c80 con 0x7f84d809af10 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.962+0000 7f84e0274640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f84d80d0a30 con 0x7f84d8071180 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84de811640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f84d809af10 0x7f84d80cdc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51814/0 (socket says 172.21.15.55:51814) 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84de811640 1 -- 172.21.15.55:0/4241071965 learned_addr learned my addr 172.21.15.55:0/4241071965 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84dd80f640 1 --2- 172.21.15.55:0/4241071965 >> v2:172.21.15.12:3300/0 conn(0x7f84d8071180 0x7f84d809a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84dd00e640 1 -- 172.21.15.55:0/4241071965 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2498522410 0 0) 0x7f84d80d0c80 con 0x7f84d809af10 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84dd00e640 1 -- 172.21.15.55:0/4241071965 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84c4003520 con 0x7f84d809af10 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84dd80f640 1 -- 172.21.15.55:0/4241071965 >> v1:172.21.15.55:6789/0 conn(0x7f84d809af10 legacy=0x7f84d80cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84dd80f640 1 -- 172.21.15.55:0/4241071965 >> v1:172.21.15.12:6789/0 conn(0x7f84d815ff10 legacy=0x7f84d809b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84dd00e640 1 -- 172.21.15.55:0/4241071965 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2915865527 0 0) 0x7f84d80716f0 con 0x7f84d815ff10 2024-04-15T15:20:46.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.963+0000 7f84dd80f640 1 -- 172.21.15.55:0/4241071965 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d80d0f60 con 0x7f84d8071180 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.964+0000 7f84dd80f640 1 --2- 172.21.15.55:0/4241071965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d8071180 0x7f84d809a9d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f84c80097a0 tx=0x7f84c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef3cfaa95bf9e5e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.964+0000 7f84dd00e640 1 -- 172.21.15.55:0/4241071965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c80315a0 con 0x7f84d8071180 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.964+0000 7f84dd00e640 1 -- 172.21.15.55:0/4241071965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c8037070 con 0x7f84d8071180 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.964+0000 7f84dd00e640 1 -- 172.21.15.55:0/4241071965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8031960 con 0x7f84d8071180 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.965+0000 7f84e0274640 1 -- 172.21.15.55:0/4241071965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d8071180 msgr2=0x7f84d809a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.965+0000 7f84e0274640 1 --2- 172.21.15.55:0/4241071965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d8071180 0x7f84d809a9d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f84c80097a0 tx=0x7f84c802b840 comp rx=0 tx=0).stop 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.965+0000 7f84e0274640 1 -- 172.21.15.55:0/4241071965 shutdown_connections 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.965+0000 7f84e0274640 1 --2- 172.21.15.55:0/4241071965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d8071180 0x7f84d809a9d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.965+0000 7f84e0274640 1 -- 172.21.15.55:0/4241071965 >> 172.21.15.55:0/4241071965 conn(0x7f84d80b3f40 msgr2=0x7f84d80b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.965+0000 7f84e0274640 1 -- 172.21.15.55:0/4241071965 shutdown_connections 2024-04-15T15:20:46.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.965+0000 7f84e0274640 1 -- 172.21.15.55:0/4241071965 wait complete. 2024-04-15T15:20:46.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.966+0000 7f84e0274640 1 Processor -- start 2024-04-15T15:20:46.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.966+0000 7f84e0274640 1 -- start start 2024-04-15T15:20:46.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84e0274640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84d8071180 0x7f84d81f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84e0274640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84d809af10 0x7f84d80c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84e0274640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d815ff10 0x7f84d815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:46.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84e0274640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f84d80ce340 con 0x7f84d815ff10 2024-04-15T15:20:46.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84e0274640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f84d80af140 con 0x7f84d8071180 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84e0274640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f84d80ce4c0 con 0x7f84d809af10 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84d8071180 0x7f84d81f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84d8071180 0x7f84d81f5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34694/0 (socket says 172.21.15.55:34694) 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84dd80f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84d809af10 0x7f84d80c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 -- 172.21.15.55:0/642809971 learned_addr learned my addr 172.21.15.55:0/642809971 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de811640 1 --2- 172.21.15.55:0/642809971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d815ff10 0x7f84d815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 -- 172.21.15.55:0/642809971 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84d809af10 msgr2=0x7f84d80c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 --2- 172.21.15.55:0/642809971 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84d809af10 0x7f84d80c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 -- 172.21.15.55:0/642809971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d815ff10 msgr2=0x7f84d815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 --2- 172.21.15.55:0/642809971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d815ff10 0x7f84d815ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 -- 172.21.15.55:0/642809971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d80c9300 con 0x7f84d8071180 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84de010640 1 --2- 172.21.15.55:0/642809971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84d8071180 0x7f84d81f5a80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f84d400c410 tx=0x7f84d400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:46.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84ceffd640 1 -- 172.21.15.55:0/642809971 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4017070 con 0x7f84d8071180 2024-04-15T15:20:46.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84ceffd640 1 -- 172.21.15.55:0/642809971 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d40035f0 con 0x7f84d8071180 2024-04-15T15:20:46.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84ceffd640 1 -- 172.21.15.55:0/642809971 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40039e0 con 0x7f84d8071180 2024-04-15T15:20:46.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.967+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d81f6900 con 0x7f84d8071180 2024-04-15T15:20:46.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.968+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d81f6340 con 0x7f84d8071180 2024-04-15T15:20:46.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.968+0000 7f84ceffd640 1 -- 172.21.15.55:0/642809971 <== mon.1 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f84d401a070 con 0x7f84d8071180 2024-04-15T15:20:46.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.968+0000 7f84ceffd640 1 -- 172.21.15.55:0/642809971 <== mon.1 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f84d4012740 con 0x7f84d8071180 2024-04-15T15:20:46.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.968+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d80cfe30 con 0x7f84d8071180 2024-04-15T15:20:46.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:46.971+0000 7f84ceffd640 1 -- 172.21.15.55:0/642809971 <== mon.1 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f84d80cfe30 con 0x7f84d8071180 2024-04-15T15:20:47.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.045+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9af3971f-5436-411b-bd33-a0ec8913f792", "id": 0} v 0) -- 0x7f84d80690f0 con 0x7f84d8071180 2024-04-15T15:20:47.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.050+0000 7f84ceffd640 1 -- 172.21.15.55:0/642809971 <== mon.1 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9af3971f-5436-411b-bd33-a0ec8913f792", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f84d4021020 con 0x7f84d8071180 2024-04-15T15:20:47.051 INFO:teuthology.orchestra.run.smithi055.stdout:0 2024-04-15T15:20:47.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84d8071180 msgr2=0x7f84d81f5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 --2- 172.21.15.55:0/642809971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84d8071180 0x7f84d81f5a80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f84d400c410 tx=0x7f84d400c8d0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 shutdown_connections 2024-04-15T15:20:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 --2- 172.21.15.55:0/642809971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84d815ff10 0x7f84d815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 --2- 172.21.15.55:0/642809971 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84d809af10 0x7f84d80c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 --2- 172.21.15.55:0/642809971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84d8071180 0x7f84d81f5a80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 >> 172.21.15.55:0/642809971 conn(0x7f84d80b3f40 msgr2=0x7f84d80b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 shutdown_connections 2024-04-15T15:20:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.051+0000 7f84e0274640 1 -- 172.21.15.55:0/642809971 wait complete. 2024-04-15T15:20:47.063 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd new 1235e7b5-5133-476f-b84d-315a337c9cd2 1 2024-04-15T15:20:47.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.165+0000 7ff155f7c640 1 Processor -- start 2024-04-15T15:20:47.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.165+0000 7ff155f7c640 1 -- start start 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.165+0000 7ff155f7c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff15006d100 0x7ff15006e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.165+0000 7ff155f7c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff15006eae0 0x7ff1500b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.165+0000 7ff155f7c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15006ef70 con 0x7ff15015e6f0 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.165+0000 7ff155f7c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff15006f220 con 0x7ff15006d100 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.165+0000 7ff155f7c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff15006f500 con 0x7ff15006eae0 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.166+0000 7ff14ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff15006eae0 0x7ff1500b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.166+0000 7ff14ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff15006eae0 0x7ff1500b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34698/0 (socket says 172.21.15.55:34698) 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.166+0000 7ff14ffff640 1 -- 172.21.15.55:0/2091328261 learned_addr learned my addr 172.21.15.55:0/2091328261 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.166+0000 7ff146ffd640 1 --2- 172.21.15.55:0/2091328261 >> v2:172.21.15.12:3300/0 conn(0x7ff15006d100 0x7ff15006e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.166+0000 7ff14ffff640 1 -- 172.21.15.55:0/2091328261 >> v2:172.21.15.12:3300/0 conn(0x7ff15006d100 msgr2=0x7ff15006e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:47.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.166+0000 7ff14ffff640 1 --2- 172.21.15.55:0/2091328261 >> v2:172.21.15.12:3300/0 conn(0x7ff15006d100 0x7ff15006e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.166+0000 7ff14ffff640 1 -- 172.21.15.55:0/2091328261 >> v1:172.21.15.55:6789/0 conn(0x7ff15015e6f0 legacy=0x7ff15006c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.166+0000 7ff14ffff640 1 -- 172.21.15.55:0/2091328261 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1500b45d0 con 0x7ff15006eae0 2024-04-15T15:20:47.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff14ffff640 1 --2- 172.21.15.55:0/2091328261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff1500b4dd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff1500cd400 tx=0x7ff14802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab48b656b855b563 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:47.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff14effd640 1 -- 172.21.15.55:0/2091328261 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14802cb20 con 0x7ff15006eae0 2024-04-15T15:20:47.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff14effd640 1 -- 172.21.15.55:0/2091328261 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff14802ccc0 con 0x7ff15006eae0 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff14effd640 1 -- 172.21.15.55:0/2091328261 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148034610 con 0x7ff15006eae0 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff155f7c640 1 -- 172.21.15.55:0/2091328261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 msgr2=0x7ff1500b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff155f7c640 1 --2- 172.21.15.55:0/2091328261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff1500b4dd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff1500cd400 tx=0x7ff14802bf80 comp rx=0 tx=0).stop 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff155f7c640 1 -- 172.21.15.55:0/2091328261 shutdown_connections 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff155f7c640 1 --2- 172.21.15.55:0/2091328261 >> v2:172.21.15.12:3300/0 conn(0x7ff15006d100 0x7ff15006e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff155f7c640 1 --2- 172.21.15.55:0/2091328261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff1500b4dd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.167+0000 7ff155f7c640 1 -- 172.21.15.55:0/2091328261 >> 172.21.15.55:0/2091328261 conn(0x7ff150072440 msgr2=0x7ff150072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.168+0000 7ff155f7c640 1 -- 172.21.15.55:0/2091328261 shutdown_connections 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.168+0000 7ff155f7c640 1 -- 172.21.15.55:0/2091328261 wait complete. 2024-04-15T15:20:47.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.169+0000 7ff155f7c640 1 Processor -- start 2024-04-15T15:20:47.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.169+0000 7ff155f7c640 1 -- start start 2024-04-15T15:20:47.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff155f7c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff15006d100 0x7ff15014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff155f7c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff15015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff155f7c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15015e6f0 0x7ff15006c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff155f7c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff1500b4370 con 0x7ff15015e6f0 2024-04-15T15:20:47.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff155f7c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff1500b41f0 con 0x7ff15006eae0 2024-04-15T15:20:47.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff155f7c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff15015ead0 con 0x7ff15006d100 2024-04-15T15:20:47.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff15015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff14f7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff15006d100 0x7ff15014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff15015a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34704/0 (socket says 172.21.15.55:34704) 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 -- 172.21.15.55:0/1911279851 learned_addr learned my addr 172.21.15.55:0/1911279851 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff14ffff640 1 --2- 172.21.15.55:0/1911279851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15015e6f0 0x7ff15006c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 -- 172.21.15.55:0/1911279851 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff15006d100 msgr2=0x7ff15014b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 --2- 172.21.15.55:0/1911279851 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff15006d100 0x7ff15014b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 -- 172.21.15.55:0/1911279851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15015e6f0 msgr2=0x7ff15006c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 --2- 172.21.15.55:0/1911279851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15015e6f0 0x7ff15006c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 -- 172.21.15.55:0/1911279851 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff150155ac0 con 0x7ff15006eae0 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff146ffd640 1 --2- 172.21.15.55:0/1911279851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff15015a4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff13800e870 tx=0x7ff13800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff14cff9640 1 -- 172.21.15.55:0/1911279851 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138017070 con 0x7ff15006eae0 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff14cff9640 1 -- 172.21.15.55:0/1911279851 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff13800c940 con 0x7ff15006eae0 2024-04-15T15:20:47.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff150155da0 con 0x7ff15006eae0 2024-04-15T15:20:47.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff14cff9640 1 -- 172.21.15.55:0/1911279851 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13800cd30 con 0x7ff15006eae0 2024-04-15T15:20:47.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.170+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff150156360 con 0x7ff15006eae0 2024-04-15T15:20:47.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.171+0000 7ff14cff9640 1 -- 172.21.15.55:0/1911279851 <== mon.1 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff13801a070 con 0x7ff15006eae0 2024-04-15T15:20:47.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.171+0000 7ff14cff9640 1 -- 172.21.15.55:0/1911279851 <== mon.1 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7ff138013470 con 0x7ff15006eae0 2024-04-15T15:20:47.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.171+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1500690f0 con 0x7ff15006eae0 2024-04-15T15:20:47.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.173+0000 7ff14cff9640 1 -- 172.21.15.55:0/1911279851 <== mon.1 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff138006020 con 0x7ff15006eae0 2024-04-15T15:20:47.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.247+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1235e7b5-5133-476f-b84d-315a337c9cd2", "id": 1} v 0) -- 0x7ff15006fd30 con 0x7ff15006eae0 2024-04-15T15:20:47.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.255+0000 7ff14cff9640 1 -- 172.21.15.55:0/1911279851 <== mon.1 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1235e7b5-5133-476f-b84d-315a337c9cd2", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7ff138005990 con 0x7ff15006eae0 2024-04-15T15:20:47.257 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-04-15T15:20:47.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 msgr2=0x7ff15015a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 --2- 172.21.15.55:0/1911279851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff15015a4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff13800e870 tx=0x7ff13800ed30 comp rx=0 tx=0).stop 2024-04-15T15:20:47.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 shutdown_connections 2024-04-15T15:20:47.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 --2- 172.21.15.55:0/1911279851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15015e6f0 0x7ff15006c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 --2- 172.21.15.55:0/1911279851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff15006eae0 0x7ff15015a4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 --2- 172.21.15.55:0/1911279851 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff15006d100 0x7ff15014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 >> 172.21.15.55:0/1911279851 conn(0x7ff150072440 msgr2=0x7ff1500b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 shutdown_connections 2024-04-15T15:20:47.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.256+0000 7ff155f7c640 1 -- 172.21.15.55:0/1911279851 wait complete. 2024-04-15T15:20:47.269 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd new b7f44bbb-0e6e-4acd-8218-14a0c31002bb 2 2024-04-15T15:20:47.354 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.352+0000 7fc6f7041040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T15:20:47.371 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.369+0000 7fb2824da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T15:20:47.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.371+0000 7f45c8540640 1 Processor -- start 2024-04-15T15:20:47.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.371+0000 7f45c8540640 1 -- start start 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c8540640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45c00b0850 0x7f45c00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c8540640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c00aeed0 0x7f45c00af360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c8540640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c015ebc0 con 0x7f45c006b9e0 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c8540640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f45c00ae150 con 0x7f45c00b0850 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c8540640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f45c0070c80 con 0x7f45c00aeed0 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c6add640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c00aeed0 0x7f45c00af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c6add640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c00aeed0 0x7f45c00af360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.55:40382/0 (socket says 172.21.15.55:40382) 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c6add640 1 -- 172.21.15.55:0/3054639801 learned_addr learned my addr 172.21.15.55:0/3054639801 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c5adb640 1 --2- 172.21.15.55:0/3054639801 >> v2:172.21.15.12:3300/0 conn(0x7f45c00b0850 0x7f45c00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c6add640 1 -- 172.21.15.55:0/3054639801 >> v2:172.21.15.12:3300/0 conn(0x7f45c00b0850 msgr2=0x7f45c00705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c6add640 1 --2- 172.21.15.55:0/3054639801 >> v2:172.21.15.12:3300/0 conn(0x7f45c00b0850 0x7f45c00705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.372+0000 7f45c6add640 1 -- 172.21.15.55:0/3054639801 >> v1:172.21.15.12:6789/0 conn(0x7f45c006b9e0 legacy=0x7f45c006bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.373+0000 7f45c6add640 1 -- 172.21.15.55:0/3054639801 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c00ae3f0 con 0x7f45c00aeed0 2024-04-15T15:20:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.373+0000 7f45c6add640 1 --2- 172.21.15.55:0/3054639801 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c00aeed0 0x7f45c00af360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009e80 tx=0x7f45b802bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bff7ccad4b85f3d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.373+0000 7f45c52da640 1 -- 172.21.15.55:0/3054639801 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b802caf0 con 0x7f45c00aeed0 2024-04-15T15:20:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.373+0000 7f45c52da640 1 -- 172.21.15.55:0/3054639801 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b802cc90 con 0x7f45c00aeed0 2024-04-15T15:20:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.373+0000 7f45c52da640 1 -- 172.21.15.55:0/3054639801 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8034610 con 0x7f45c00aeed0 2024-04-15T15:20:47.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.374+0000 7f45c8540640 1 -- 172.21.15.55:0/3054639801 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c00aeed0 msgr2=0x7f45c00af360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.374+0000 7f45c8540640 1 --2- 172.21.15.55:0/3054639801 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c00aeed0 0x7f45c00af360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009e80 tx=0x7f45b802bf50 comp rx=0 tx=0).stop 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.374+0000 7f45c8540640 1 -- 172.21.15.55:0/3054639801 shutdown_connections 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.374+0000 7f45c8540640 1 --2- 172.21.15.55:0/3054639801 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c00aeed0 0x7f45c00af360 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.374+0000 7f45c8540640 1 --2- 172.21.15.55:0/3054639801 >> v2:172.21.15.12:3300/0 conn(0x7f45c00b0850 0x7f45c00705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.374+0000 7f45c8540640 1 -- 172.21.15.55:0/3054639801 >> 172.21.15.55:0/3054639801 conn(0x7f45c00b2e70 msgr2=0x7f45c00b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.374+0000 7f45c8540640 1 -- 172.21.15.55:0/3054639801 shutdown_connections 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.374+0000 7f45c8540640 1 -- 172.21.15.55:0/3054639801 wait complete. 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.375+0000 7f45c8540640 1 Processor -- start 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.375+0000 7f45c8540640 1 -- start start 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c8540640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c006b9e0 0x7f45c01f8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c8540640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c00aeed0 0x7f45c0149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c8540640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c00b0850 0x7f45c00b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c8540640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f45c00cf0a0 con 0x7f45c00b0850 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c8540640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f45c00ae9b0 con 0x7f45c00aeed0 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c8540640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f45c00af8a0 con 0x7f45c006b9e0 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c5adb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c00aeed0 0x7f45c0149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c62dc640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c006b9e0 0x7f45c01f8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c5adb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c00aeed0 0x7f45c0149040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34720/0 (socket says 172.21.15.55:34720) 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c5adb640 1 -- 172.21.15.55:0/3678035279 learned_addr learned my addr 172.21.15.55:0/3678035279 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.376+0000 7f45c6add640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c00b0850 0x7f45c00b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c62dc640 1 -- 172.21.15.55:0/3678035279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c00aeed0 msgr2=0x7f45c0149040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c62dc640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c00aeed0 0x7f45c0149040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c62dc640 1 -- 172.21.15.55:0/3678035279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c00b0850 msgr2=0x7f45c00b5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c62dc640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c00b0850 0x7f45c00b5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c62dc640 1 -- 172.21.15.55:0/3678035279 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c0149580 con 0x7f45c006b9e0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c5adb640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c00aeed0 0x7f45c0149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c62dc640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c006b9e0 0x7f45c01f8730 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00c340 tx=0x7f45bc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45b2ffd640 1 -- 172.21.15.55:0/3678035279 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0052d0 con 0x7f45c006b9e0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45b2ffd640 1 -- 172.21.15.55:0/3678035279 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45bc005490 con 0x7f45c006b9e0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45b2ffd640 1 -- 172.21.15.55:0/3678035279 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc007260 con 0x7f45c006b9e0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c00b6680 con 0x7f45c006b9e0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c00b6c40 con 0x7f45c006b9e0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45b2ffd640 1 -- 172.21.15.55:0/3678035279 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f45bc007400 con 0x7f45c006b9e0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.377+0000 7f45b2ffd640 1 -- 172.21.15.55:0/3678035279 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f45bc011590 con 0x7f45c006b9e0 2024-04-15T15:20:47.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.378+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c00690f0 con 0x7f45c006b9e0 2024-04-15T15:20:47.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.381+0000 7f45b2ffd640 1 -- 172.21.15.55:0/3678035279 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f45bc003a50 con 0x7f45c006b9e0 2024-04-15T15:20:47.412 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.410+0000 7fc6f7041040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T15:20:47.428 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.426+0000 7fb2824da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T15:20:47.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.456+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b7f44bbb-0e6e-4acd-8218-14a0c31002bb", "id": 2} v 0) -- 0x7f45c006dd00 con 0x7f45c006b9e0 2024-04-15T15:20:47.458 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.457+0000 7fc6f7041040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T15:20:47.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.461+0000 7f45b2ffd640 1 -- 172.21.15.55:0/3678035279 <== mon.2 v2:172.21.15.55:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7f44bbb-0e6e-4acd-8218-14a0c31002bb", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f45bc003610 con 0x7f45c006b9e0 2024-04-15T15:20:47.463 INFO:teuthology.orchestra.run.smithi055.stdout:2 2024-04-15T15:20:47.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c006b9e0 msgr2=0x7f45c01f8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c006b9e0 0x7f45c01f8730 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00c340 tx=0x7f45bc00c800 comp rx=0 tx=0).stop 2024-04-15T15:20:47.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 shutdown_connections 2024-04-15T15:20:47.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45c00b0850 0x7f45c00b5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c00aeed0 0x7f45c0149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 --2- 172.21.15.55:0/3678035279 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45c006b9e0 0x7f45c01f8730 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 >> 172.21.15.55:0/3678035279 conn(0x7f45c00b2e70 msgr2=0x7f45c006ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 shutdown_connections 2024-04-15T15:20:47.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.462+0000 7f45c8540640 1 -- 172.21.15.55:0/3678035279 wait complete. 2024-04-15T15:20:47.474 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd new 76b01b05-bf0f-4054-affd-821212dcd1fd 3 2024-04-15T15:20:47.475 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.473+0000 7fb2824da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T15:20:47.518 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.516+0000 7fc6f7041040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T15:20:47.531 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.529+0000 7fb2824da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T15:20:47.562 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.561+0000 7fc6f7041040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T15:20:47.575 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.573+0000 7fb2824da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T15:20:47.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.577+0000 7f09927d4640 1 Processor -- start 2024-04-15T15:20:47.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.577+0000 7f09927d4640 1 -- start start 2024-04-15T15:20:47.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f09927d4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f098c0ceee0 0x7f098c0ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f09927d4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f098c0b08e0 0x7f098c0cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f09927d4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f098c0cd9f0 con 0x7f098c06b9e0 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f09927d4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f098c0ce000 con 0x7f098c0ceee0 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f09927d4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f098c0709f0 con 0x7f098c0b08e0 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f0990d71640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f098c0b08e0 0x7f098c0cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f0990d71640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f098c0b08e0 0x7f098c0cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34722/0 (socket says 172.21.15.55:34722) 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f0990d71640 1 -- 172.21.15.55:0/542717798 learned_addr learned my addr 172.21.15.55:0/542717798 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f098b7fe640 1 --2- 172.21.15.55:0/542717798 >> v2:172.21.15.12:3300/0 conn(0x7f098c0ceee0 0x7f098c0ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.578+0000 7f0990d71640 1 -- 172.21.15.55:0/542717798 >> v2:172.21.15.12:3300/0 conn(0x7f098c0ceee0 msgr2=0x7f098c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.579+0000 7f0990d71640 1 --2- 172.21.15.55:0/542717798 >> v2:172.21.15.12:3300/0 conn(0x7f098c0ceee0 0x7f098c0ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.579+0000 7f0990d71640 1 -- 172.21.15.55:0/542717798 >> v1:172.21.15.12:6789/0 conn(0x7f098c06b9e0 legacy=0x7f098c06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.579+0000 7f0990d71640 1 -- 172.21.15.55:0/542717798 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f098c0cdc90 con 0x7f098c0b08e0 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.579+0000 7f0990d71640 1 --2- 172.21.15.55:0/542717798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0b08e0 0x7f098c0cb8f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f098c06ea00 tx=0x7f098402c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97065479e5113fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.579+0000 7f098affd640 1 -- 172.21.15.55:0/542717798 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098402cbb0 con 0x7f098c0b08e0 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.579+0000 7f098affd640 1 -- 172.21.15.55:0/542717798 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098402cd50 con 0x7f098c0b08e0 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.579+0000 7f098affd640 1 -- 172.21.15.55:0/542717798 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0984034610 con 0x7f098c0b08e0 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.580+0000 7f09927d4640 1 -- 172.21.15.55:0/542717798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0b08e0 msgr2=0x7f098c0cb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.580+0000 7f09927d4640 1 --2- 172.21.15.55:0/542717798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0b08e0 0x7f098c0cb8f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f098c06ea00 tx=0x7f098402c010 comp rx=0 tx=0).stop 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.580+0000 7f09927d4640 1 -- 172.21.15.55:0/542717798 shutdown_connections 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.580+0000 7f09927d4640 1 --2- 172.21.15.55:0/542717798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0b08e0 0x7f098c0cb8f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.580+0000 7f09927d4640 1 --2- 172.21.15.55:0/542717798 >> v2:172.21.15.12:3300/0 conn(0x7f098c0ceee0 0x7f098c0ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.580+0000 7f09927d4640 1 -- 172.21.15.55:0/542717798 >> 172.21.15.55:0/542717798 conn(0x7f098c0b2e70 msgr2=0x7f098c0b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.580+0000 7f09927d4640 1 -- 172.21.15.55:0/542717798 shutdown_connections 2024-04-15T15:20:47.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.580+0000 7f09927d4640 1 -- 172.21.15.55:0/542717798 wait complete. 2024-04-15T15:20:47.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.582+0000 7f09927d4640 1 Processor -- start 2024-04-15T15:20:47.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.583+0000 7f09927d4640 1 -- start start 2024-04-15T15:20:47.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f09927d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f098c06b9e0 0x7f098c157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f09927d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f098c0b08e0 0x7f098c1f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f09927d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0ceee0 0x7f098c0c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f09927d4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f098c0ce9d0 con 0x7f098c0b08e0 2024-04-15T15:20:47.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f09927d4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f098c0705a0 con 0x7f098c0ceee0 2024-04-15T15:20:47.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f09927d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f098c1f4c50 con 0x7f098c06b9e0 2024-04-15T15:20:47.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098bfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f098c06b9e0 0x7f098c157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f0990d71640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0ceee0 0x7f098c0c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f098c0b08e0 0x7f098c1f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f098c0b08e0 0x7f098c1f4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.55:55996/0 (socket says 172.21.15.55:55996) 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098bfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f098c06b9e0 0x7f098c157be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.55:40400/0 (socket says 172.21.15.55:40400) 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f0990d71640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0ceee0 0x7f098c0c6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34728/0 (socket says 172.21.15.55:34728) 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098b7fe640 1 -- 172.21.15.55:0/1757331105 learned_addr learned my addr 172.21.15.55:0/1757331105 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098bfff640 1 -- 172.21.15.55:0/1757331105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0ceee0 msgr2=0x7f098c0c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098bfff640 1 --2- 172.21.15.55:0/1757331105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0ceee0 0x7f098c0c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098bfff640 1 -- 172.21.15.55:0/1757331105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f098c0b08e0 msgr2=0x7f098c1f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098bfff640 1 --2- 172.21.15.55:0/1757331105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f098c0b08e0 0x7f098c1f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098bfff640 1 -- 172.21.15.55:0/1757331105 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f098c1f4eb0 con 0x7f098c06b9e0 2024-04-15T15:20:47.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.584+0000 7f098b7fe640 1 --2- 172.21.15.55:0/1757331105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f098c0b08e0 0x7f098c1f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.585+0000 7f098bfff640 1 --2- 172.21.15.55:0/1757331105 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f098c06b9e0 0x7f098c157be0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f097c00c3c0 tx=0x7f097c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.585+0000 7f0988ff9640 1 -- 172.21.15.55:0/1757331105 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c0052d0 con 0x7f098c06b9e0 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.585+0000 7f0988ff9640 1 -- 172.21.15.55:0/1757331105 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f097c005490 con 0x7f098c06b9e0 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.585+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f098c1f5190 con 0x7f098c06b9e0 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.585+0000 7f0988ff9640 1 -- 172.21.15.55:0/1757331105 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c007260 con 0x7f098c06b9e0 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.585+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f098c0c4fe0 con 0x7f098c06b9e0 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.585+0000 7f0988ff9640 1 -- 172.21.15.55:0/1757331105 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f097c007400 con 0x7f098c06b9e0 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.585+0000 7f0988ff9640 1 -- 172.21.15.55:0/1757331105 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f097c011590 con 0x7f098c06b9e0 2024-04-15T15:20:47.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.586+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f098c0c4dd0 con 0x7f098c06b9e0 2024-04-15T15:20:47.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.590+0000 7f0988ff9640 1 -- 172.21.15.55:0/1757331105 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f097c0035f0 con 0x7f098c06b9e0 2024-04-15T15:20:47.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.664+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "osd new", "uuid": "76b01b05-bf0f-4054-affd-821212dcd1fd", "id": 3} v 0) -- 0x7f098c0690f0 con 0x7f098c06b9e0 2024-04-15T15:20:47.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.674+0000 7f0988ff9640 1 -- 172.21.15.55:0/1757331105 <== mon.2 v2:172.21.15.55:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76b01b05-bf0f-4054-affd-821212dcd1fd", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f097c003b70 con 0x7f098c06b9e0 2024-04-15T15:20:47.676 INFO:teuthology.orchestra.run.smithi055.stdout:3 2024-04-15T15:20:47.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f098c06b9e0 msgr2=0x7f098c157be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 --2- 172.21.15.55:0/1757331105 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f098c06b9e0 0x7f098c157be0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f097c00c3c0 tx=0x7f097c00c880 comp rx=0 tx=0).stop 2024-04-15T15:20:47.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 shutdown_connections 2024-04-15T15:20:47.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 --2- 172.21.15.55:0/1757331105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f098c0ceee0 0x7f098c0c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 --2- 172.21.15.55:0/1757331105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f098c0b08e0 0x7f098c1f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 --2- 172.21.15.55:0/1757331105 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f098c06b9e0 0x7f098c157be0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 >> 172.21.15.55:0/1757331105 conn(0x7f098c0b2e70 msgr2=0x7f098c15dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 shutdown_connections 2024-04-15T15:20:47.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.675+0000 7f09927d4640 1 -- 172.21.15.55:0/1757331105 wait complete. 2024-04-15T15:20:47.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd new 824cae86-45ab-4f14-a657-0ac0235573aa 4 2024-04-15T15:20:47.691 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.690+0000 7fc6f7041040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T15:20:47.698 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.696+0000 7fb2824da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T15:20:47.766 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.764+0000 7fc6f7041040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T15:20:47.771 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.769+0000 7fb2824da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T15:20:47.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.793+0000 7f87e177c640 1 Processor -- start 2024-04-15T15:20:47.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.793+0000 7f87e177c640 1 -- start start 2024-04-15T15:20:47.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87e177c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f87dc15e700 0x7f87dc0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87e177c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc0d2360 0x7f87dc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87e177c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87dc0b47a0 con 0x7f87dc15eae0 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87e177c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f87dc0b53b0 con 0x7f87dc15e700 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87e177c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f87dc0b4a80 con 0x7f87dc0d2360 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87d3fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc0d2360 0x7f87dc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87d3fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc0d2360 0x7f87dc0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.55:40402/0 (socket says 172.21.15.55:40402) 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87d3fff640 1 -- 172.21.15.55:0/365653798 learned_addr learned my addr 172.21.15.55:0/365653798 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87daffd640 1 --2- 172.21.15.55:0/365653798 >> v2:172.21.15.12:3300/0 conn(0x7f87dc15e700 0x7f87dc0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87d3fff640 1 -- 172.21.15.55:0/365653798 >> v2:172.21.15.12:3300/0 conn(0x7f87dc15e700 msgr2=0x7f87dc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87d3fff640 1 --2- 172.21.15.55:0/365653798 >> v2:172.21.15.12:3300/0 conn(0x7f87dc15e700 0x7f87dc0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87d3fff640 1 -- 172.21.15.55:0/365653798 >> v1:172.21.15.12:6789/0 conn(0x7f87dc15eae0 legacy=0x7f87dc0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.794+0000 7f87d3fff640 1 -- 172.21.15.55:0/365653798 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc0b4d60 con 0x7f87dc0d2360 2024-04-15T15:20:47.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.795+0000 7f87d3fff640 1 --2- 172.21.15.55:0/365653798 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc0d2360 0x7f87dc0d2b60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009870 tx=0x7f87c402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8cfd5e6668c269a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:47.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.795+0000 7f87da7fc640 1 -- 172.21.15.55:0/365653798 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c4034070 con 0x7f87dc0d2360 2024-04-15T15:20:47.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.795+0000 7f87da7fc640 1 -- 172.21.15.55:0/365653798 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c4030840 con 0x7f87dc0d2360 2024-04-15T15:20:47.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.795+0000 7f87da7fc640 1 -- 172.21.15.55:0/365653798 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c4030c10 con 0x7f87dc0d2360 2024-04-15T15:20:47.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.796+0000 7f87e177c640 1 -- 172.21.15.55:0/365653798 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc0d2360 msgr2=0x7f87dc0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.796+0000 7f87e177c640 1 --2- 172.21.15.55:0/365653798 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc0d2360 0x7f87dc0d2b60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009870 tx=0x7f87c402b980 comp rx=0 tx=0).stop 2024-04-15T15:20:47.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.796+0000 7f87e177c640 1 -- 172.21.15.55:0/365653798 shutdown_connections 2024-04-15T15:20:47.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.796+0000 7f87e177c640 1 --2- 172.21.15.55:0/365653798 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc0d2360 0x7f87dc0d2b60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.796+0000 7f87e177c640 1 --2- 172.21.15.55:0/365653798 >> v2:172.21.15.12:3300/0 conn(0x7f87dc15e700 0x7f87dc0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.796+0000 7f87e177c640 1 -- 172.21.15.55:0/365653798 >> 172.21.15.55:0/365653798 conn(0x7f87dc08bf60 msgr2=0x7f87dc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.796+0000 7f87e177c640 1 -- 172.21.15.55:0/365653798 shutdown_connections 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.796+0000 7f87e177c640 1 -- 172.21.15.55:0/365653798 wait complete. 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.797+0000 7f87e177c640 1 Processor -- start 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.797+0000 7f87e177c640 1 -- start start 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87e177c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc0d2360 0x7f87dc159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87e177c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87dc15e700 0x7f87dc161330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87e177c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc15eae0 0x7f87dc161870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87e177c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc0b5d80 con 0x7f87dc0d2360 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87e177c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc0b4450 con 0x7f87dc15e700 2024-04-15T15:20:47.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87e177c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f87dc0d1d40 con 0x7f87dc15eae0 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc15eae0 0x7f87dc161870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc15eae0 0x7f87dc161870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.55:40414/0 (socket says 172.21.15.55:40414) 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 -- 172.21.15.55:0/2729780476 learned_addr learned my addr 172.21.15.55:0/2729780476 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87d3fff640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87dc15e700 0x7f87dc161330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87daffd640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc0d2360 0x7f87dc159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 -- 172.21.15.55:0/2729780476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87dc15e700 msgr2=0x7f87dc161330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87dc15e700 0x7f87dc161330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 -- 172.21.15.55:0/2729780476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc0d2360 msgr2=0x7f87dc159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc0d2360 0x7f87dc159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 -- 172.21.15.55:0/2729780476 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc159c80 con 0x7f87dc15eae0 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87db7fe640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc15eae0 0x7f87dc161870 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c6e0 tx=0x7f87d400cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87d37fe640 1 -- 172.21.15.55:0/2729780476 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4003b70 con 0x7f87dc15eae0 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.798+0000 7f87d37fe640 1 -- 172.21.15.55:0/2729780476 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87d4003d30 con 0x7f87dc15eae0 2024-04-15T15:20:47.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.799+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87dc0b9ec0 con 0x7f87dc15eae0 2024-04-15T15:20:47.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.799+0000 7f87d37fe640 1 -- 172.21.15.55:0/2729780476 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4005310 con 0x7f87dc15eae0 2024-04-15T15:20:47.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.799+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87dc0b8050 con 0x7f87dc15eae0 2024-04-15T15:20:47.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.799+0000 7f87d37fe640 1 -- 172.21.15.55:0/2729780476 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f87d40054b0 con 0x7f87dc15eae0 2024-04-15T15:20:47.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.799+0000 7f87d37fe640 1 -- 172.21.15.55:0/2729780476 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f87d40057a0 con 0x7f87dc15eae0 2024-04-15T15:20:47.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.799+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87dc0b7b90 con 0x7f87dc15eae0 2024-04-15T15:20:47.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.801+0000 7f87d37fe640 1 -- 172.21.15.55:0/2729780476 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f87dc0b7b90 con 0x7f87dc15eae0 2024-04-15T15:20:47.815 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.814+0000 7fc6f7041040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T15:20:47.819 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.817+0000 7fb2824da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T15:20:47.870 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.869+0000 7fc6f7041040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T15:20:47.874 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.871+0000 7fb2824da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T15:20:47.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.875+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "osd new", "uuid": "824cae86-45ab-4f14-a657-0ac0235573aa", "id": 4} v 0) -- 0x7f87dc0689e0 con 0x7f87dc15eae0 2024-04-15T15:20:47.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.880+0000 7f87d37fe640 1 -- 172.21.15.55:0/2729780476 <== mon.2 v2:172.21.15.55:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "824cae86-45ab-4f14-a657-0ac0235573aa", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f87d400a640 con 0x7f87dc15eae0 2024-04-15T15:20:47.882 INFO:teuthology.orchestra.run.smithi055.stdout:4 2024-04-15T15:20:47.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc15eae0 msgr2=0x7f87dc161870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:47.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc15eae0 0x7f87dc161870 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c6e0 tx=0x7f87d400cba0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 shutdown_connections 2024-04-15T15:20:47.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f87dc15eae0 0x7f87dc161870 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87dc15e700 0x7f87dc161330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 --2- 172.21.15.55:0/2729780476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc0d2360 0x7f87dc159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:47.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 >> 172.21.15.55:0/2729780476 conn(0x7f87dc08bf60 msgr2=0x7f87dc15ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:47.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 shutdown_connections 2024-04-15T15:20:47.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.881+0000 7f87e177c640 1 -- 172.21.15.55:0/2729780476 wait complete. 2024-04-15T15:20:47.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd new cc401892-8b26-4f18-aeff-38af9be45bad 5 2024-04-15T15:20:47.922 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:47.920+0000 7fc6f7041040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T15:20:47.923 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:47.921+0000 7fb2824da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T15:20:47.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.995+0000 7f908569a640 1 Processor -- start 2024-04-15T15:20:47.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.995+0000 7f908569a640 1 -- start start 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f908569a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f908015e700 0x7f90800b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f908569a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f90800d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f908569a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f908015eae0 0x7f90800d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f908569a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f90800b53b0 con 0x7f908015eae0 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f908569a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f90800b47a0 con 0x7f908015e700 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f908569a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f90800b4a80 con 0x7f90800d2360 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907effd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f908015e700 0x7f90800b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907e7fc640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f90800d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907effd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f908015e700 0x7f90800b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34732/0 (socket says 172.21.15.55:34732) 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907effd640 1 -- 172.21.15.55:0/1145984330 learned_addr learned my addr 172.21.15.55:0/1145984330 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907f7fe640 1 --2- 172.21.15.55:0/1145984330 >> v2:172.21.15.12:3300/0 conn(0x7f908015eae0 0x7f90800d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907effd640 1 -- 172.21.15.55:0/1145984330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 msgr2=0x7f90800d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907effd640 1 --2- 172.21.15.55:0/1145984330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f90800d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907effd640 1 -- 172.21.15.55:0/1145984330 >> v2:172.21.15.12:3300/0 conn(0x7f908015eae0 msgr2=0x7f90800d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907effd640 1 --2- 172.21.15.55:0/1145984330 >> v2:172.21.15.12:3300/0 conn(0x7f908015eae0 0x7f90800d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907effd640 1 -- 172.21.15.55:0/1145984330 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90800b4d60 con 0x7f908015e700 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907f7fe640 1 --2- 172.21.15.55:0/1145984330 >> v2:172.21.15.12:3300/0 conn(0x7f908015eae0 0x7f90800d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.996+0000 7f907e7fc640 1 --2- 172.21.15.55:0/1145984330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f90800d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.997+0000 7f907effd640 1 --2- 172.21.15.55:0/1145984330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015e700 0x7f90800b5840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f908006c6c0 tx=0x7f907402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1d6a3d1bde39ae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.997+0000 7f907dffb640 1 -- 172.21.15.55:0/1145984330 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907402d040 con 0x7f908015e700 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.997+0000 7f907dffb640 1 -- 172.21.15.55:0/1145984330 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9074004230 con 0x7f908015e700 2024-04-15T15:20:48.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.997+0000 7f907dffb640 1 -- 172.21.15.55:0/1145984330 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9074003610 con 0x7f908015e700 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 -- 172.21.15.55:0/1145984330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015e700 msgr2=0x7f90800b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 --2- 172.21.15.55:0/1145984330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015e700 0x7f90800b5840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f908006c6c0 tx=0x7f907402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 -- 172.21.15.55:0/1145984330 shutdown_connections 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 --2- 172.21.15.55:0/1145984330 >> v2:172.21.15.12:3300/0 conn(0x7f908015eae0 0x7f90800d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 --2- 172.21.15.55:0/1145984330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f90800d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 --2- 172.21.15.55:0/1145984330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015e700 0x7f90800b5840 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 -- 172.21.15.55:0/1145984330 >> 172.21.15.55:0/1145984330 conn(0x7f908008bf60 msgr2=0x7f90800d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 -- 172.21.15.55:0/1145984330 shutdown_connections 2024-04-15T15:20:48.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:47.998+0000 7f908569a640 1 -- 172.21.15.55:0/1145984330 wait complete. 2024-04-15T15:20:48.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.000+0000 7f908569a640 1 Processor -- start 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.000+0000 7f908569a640 1 -- start start 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f908569a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f9080152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f908569a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908015e700 0x7f9080156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f908569a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015eae0 0x7f90801e81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f908569a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90800b5d80 con 0x7f908015e700 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f908569a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f90800b4450 con 0x7f908015eae0 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f908569a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f90800d1d40 con 0x7f90800d2360 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f907effd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f9080152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f907effd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f9080152b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.55:40430/0 (socket says 172.21.15.55:40430) 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f907f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015eae0 0x7f90801e81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f907effd640 1 -- 172.21.15.55:0/2476036523 learned_addr learned my addr 172.21.15.55:0/2476036523 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.001+0000 7f907e7fc640 1 --2- 172.21.15.55:0/2476036523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908015e700 0x7f9080156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f907effd640 1 -- 172.21.15.55:0/2476036523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015eae0 msgr2=0x7f90801e81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f907effd640 1 --2- 172.21.15.55:0/2476036523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015eae0 0x7f90801e81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f907effd640 1 -- 172.21.15.55:0/2476036523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908015e700 msgr2=0x7f9080156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f907effd640 1 --2- 172.21.15.55:0/2476036523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908015e700 0x7f9080156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f907effd640 1 -- 172.21.15.55:0/2476036523 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9080157410 con 0x7f90800d2360 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f907effd640 1 --2- 172.21.15.55:0/2476036523 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f9080152b00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f90740090b0 tx=0x7f90740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f90677fe640 1 -- 172.21.15.55:0/2476036523 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9074003c50 con 0x7f90800d2360 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f90677fe640 1 -- 172.21.15.55:0/2476036523 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9074003e10 con 0x7f90800d2360 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90801576f0 con 0x7f90800d2360 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f90677fe640 1 -- 172.21.15.55:0/2476036523 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9074006280 con 0x7f90800d2360 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9080157cb0 con 0x7f90800d2360 2024-04-15T15:20:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f90677fe640 1 -- 172.21.15.55:0/2476036523 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f907404d440 con 0x7f90800d2360 2024-04-15T15:20:48.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.002+0000 7f90677fe640 1 -- 172.21.15.55:0/2476036523 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f9074006850 con 0x7f90800d2360 2024-04-15T15:20:48.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.003+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90800689e0 con 0x7f90800d2360 2024-04-15T15:20:48.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.005+0000 7f90677fe640 1 -- 172.21.15.55:0/2476036523 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9074031440 con 0x7f90800d2360 2024-04-15T15:20:48.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.078+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cc401892-8b26-4f18-aeff-38af9be45bad", "id": 5} v 0) -- 0x7f90800ba500 con 0x7f90800d2360 2024-04-15T15:20:48.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.084+0000 7f90677fe640 1 -- 172.21.15.55:0/2476036523 <== mon.2 v2:172.21.15.55:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cc401892-8b26-4f18-aeff-38af9be45bad", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f9074031620 con 0x7f90800d2360 2024-04-15T15:20:48.086 INFO:teuthology.orchestra.run.smithi055.stdout:5 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 msgr2=0x7f9080152b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 --2- 172.21.15.55:0/2476036523 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f9080152b00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f90740090b0 tx=0x7f90740046c0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 shutdown_connections 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 --2- 172.21.15.55:0/2476036523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908015eae0 0x7f90801e81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 --2- 172.21.15.55:0/2476036523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f908015e700 0x7f9080156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 --2- 172.21.15.55:0/2476036523 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90800d2360 0x7f9080152b00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 >> 172.21.15.55:0/2476036523 conn(0x7f908008bf60 msgr2=0x7f90800d10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 shutdown_connections 2024-04-15T15:20:48.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.085+0000 7f908569a640 1 -- 172.21.15.55:0/2476036523 wait complete. 2024-04-15T15:20:48.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd new 4a9b133b-ec62-4446-8e2c-3a1903c73fec 6 2024-04-15T15:20:48.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.240+0000 7f89bf70d640 1 Processor -- start 2024-04-15T15:20:48.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.240+0000 7f89bf70d640 1 -- start start 2024-04-15T15:20:48.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bf70d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cfd60 0x7f89b80ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bf70d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b80cbee0 con 0x7f89b80cee70 2024-04-15T15:20:48.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bf70d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b80cc1c0 con 0x7f89b815b520 2024-04-15T15:20:48.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bf70d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f89b80cc4a0 con 0x7f89b80cfd60 2024-04-15T15:20:48.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bd4a9640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f89b815b520 0x7f89b80ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51824/0 (socket says 172.21.15.55:51824) 2024-04-15T15:20:48.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bd4a9640 1 -- 172.21.15.55:0/4077981839 learned_addr learned my addr 172.21.15.55:0/4077981839 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-04-15T15:20:48.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bdcaa640 1 --2- 172.21.15.55:0/4077981839 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cfd60 0x7f89b80ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89affff640 1 -- 172.21.15.55:0/4077981839 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686928838 0 0) 0x7f89b80cc1c0 con 0x7f89b815b520 2024-04-15T15:20:48.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89affff640 1 -- 172.21.15.55:0/4077981839 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a4003520 con 0x7f89b815b520 2024-04-15T15:20:48.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89affff640 1 -- 172.21.15.55:0/4077981839 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3657387798 0 0) 0x7f89b80cbee0 con 0x7f89b80cee70 2024-04-15T15:20:48.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bdcaa640 1 -- 172.21.15.55:0/4077981839 >> v1:172.21.15.55:6789/0 conn(0x7f89b815b520 legacy=0x7f89b80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bdcaa640 1 -- 172.21.15.55:0/4077981839 >> v1:172.21.15.12:6789/0 conn(0x7f89b80cee70 legacy=0x7f89b80cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.241+0000 7f89bdcaa640 1 -- 172.21.15.55:0/4077981839 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b80cc780 con 0x7f89b80cfd60 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.242+0000 7f89bdcaa640 1 --2- 172.21.15.55:0/4077981839 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cfd60 0x7f89b80ccca0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009dc0 tx=0x7f89b002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95d7d9cdc4dbc4cb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.242+0000 7f89affff640 1 -- 172.21.15.55:0/4077981839 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0039070 con 0x7f89b80cfd60 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.242+0000 7f89affff640 1 -- 172.21.15.55:0/4077981839 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b002ccd0 con 0x7f89b80cfd60 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.242+0000 7f89affff640 1 -- 172.21.15.55:0/4077981839 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0034660 con 0x7f89b80cfd60 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.243+0000 7f89bf70d640 1 -- 172.21.15.55:0/4077981839 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cfd60 msgr2=0x7f89b80ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.243+0000 7f89bf70d640 1 --2- 172.21.15.55:0/4077981839 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cfd60 0x7f89b80ccca0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009dc0 tx=0x7f89b002bf20 comp rx=0 tx=0).stop 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.243+0000 7f89bf70d640 1 -- 172.21.15.55:0/4077981839 shutdown_connections 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.243+0000 7f89bf70d640 1 --2- 172.21.15.55:0/4077981839 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cfd60 0x7f89b80ccca0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.243+0000 7f89bf70d640 1 -- 172.21.15.55:0/4077981839 >> 172.21.15.55:0/4077981839 conn(0x7f89b80b2e70 msgr2=0x7f89b80b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.243+0000 7f89bf70d640 1 -- 172.21.15.55:0/4077981839 shutdown_connections 2024-04-15T15:20:48.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.243+0000 7f89bf70d640 1 -- 172.21.15.55:0/4077981839 wait complete. 2024-04-15T15:20:48.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.244+0000 7f89bf70d640 1 Processor -- start 2024-04-15T15:20:48.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.244+0000 7f89bf70d640 1 -- start start 2024-04-15T15:20:48.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bf70d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cee70 0x7f89b80c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bf70d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89b80cfd60 0x7f89b80b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bf70d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815b520 0x7f89b80c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bf70d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f89b815b900 con 0x7f89b815b520 2024-04-15T15:20:48.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bf70d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f89b806eda0 con 0x7f89b80cfd60 2024-04-15T15:20:48.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bf70d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f89b81579f0 con 0x7f89b80cee70 2024-04-15T15:20:48.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bd4a9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cee70 0x7f89b80c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bd4a9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cee70 0x7f89b80c4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.55:40454/0 (socket says 172.21.15.55:40454) 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bd4a9640 1 -- 172.21.15.55:0/2782763545 learned_addr learned my addr 172.21.15.55:0/2782763545 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bcca8640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89b80cfd60 0x7f89b80b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bdcaa640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815b520 0x7f89b80c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bd4a9640 1 -- 172.21.15.55:0/2782763545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89b80cfd60 msgr2=0x7f89b80b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bd4a9640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89b80cfd60 0x7f89b80b6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bd4a9640 1 -- 172.21.15.55:0/2782763545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815b520 msgr2=0x7f89b80c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bd4a9640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815b520 0x7f89b80c7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.245+0000 7f89bd4a9640 1 -- 172.21.15.55:0/2782763545 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b81f2af0 con 0x7f89b80cee70 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.246+0000 7f89bd4a9640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cee70 0x7f89b80c4dd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f89a800c390 tx=0x7f89a800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.246+0000 7f89adffb640 1 -- 172.21.15.55:0/2782763545 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8017070 con 0x7f89b80cee70 2024-04-15T15:20:48.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.246+0000 7f89adffb640 1 -- 172.21.15.55:0/2782763545 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a8005ce0 con 0x7f89b80cee70 2024-04-15T15:20:48.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.246+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b81f2dd0 con 0x7f89b80cee70 2024-04-15T15:20:48.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.246+0000 7f89adffb640 1 -- 172.21.15.55:0/2782763545 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a80072f0 con 0x7f89b80cee70 2024-04-15T15:20:48.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.246+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b81f3390 con 0x7f89b80cee70 2024-04-15T15:20:48.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.246+0000 7f89adffb640 1 -- 172.21.15.55:0/2782763545 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f89a801a070 con 0x7f89b80cee70 2024-04-15T15:20:48.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.246+0000 7f89adffb640 1 -- 172.21.15.55:0/2782763545 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f89a80129f0 con 0x7f89b80cee70 2024-04-15T15:20:48.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.247+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b80690f0 con 0x7f89b80cee70 2024-04-15T15:20:48.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.249+0000 7f89adffb640 1 -- 172.21.15.55:0/2782763545 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f89a8021020 con 0x7f89b80cee70 2024-04-15T15:20:48.322 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:48.320+0000 7fb2824da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T15:20:48.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.323+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4a9b133b-ec62-4446-8e2c-3a1903c73fec", "id": 6} v 0) -- 0x7f89b8072fe0 con 0x7f89b80cee70 2024-04-15T15:20:48.327 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:48.326+0000 7fc6f7041040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T15:20:48.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.330+0000 7f89adffb640 1 -- 172.21.15.55:0/2782763545 <== mon.2 v2:172.21.15.55:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a9b133b-ec62-4446-8e2c-3a1903c73fec", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f89a80075d0 con 0x7f89b80cee70 2024-04-15T15:20:48.332 INFO:teuthology.orchestra.run.smithi055.stdout:6 2024-04-15T15:20:48.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.331+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cee70 msgr2=0x7f89b80c4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.331+0000 7f89bf70d640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cee70 0x7f89b80c4dd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f89a800c390 tx=0x7f89a800c850 comp rx=0 tx=0).stop 2024-04-15T15:20:48.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.331+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 shutdown_connections 2024-04-15T15:20:48.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.331+0000 7f89bf70d640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89b815b520 0x7f89b80c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.331+0000 7f89bf70d640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89b80cfd60 0x7f89b80b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.331+0000 7f89bf70d640 1 --2- 172.21.15.55:0/2782763545 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f89b80cee70 0x7f89b80c4dd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.331+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 >> 172.21.15.55:0/2782763545 conn(0x7f89b80b2e70 msgr2=0x7f89b80705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:48.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.332+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 shutdown_connections 2024-04-15T15:20:48.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.332+0000 7f89bf70d640 1 -- 172.21.15.55:0/2782763545 wait complete. 2024-04-15T15:20:48.343 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd new 36106542-b6bf-439a-a645-5426a655af97 7 2024-04-15T15:20:48.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.445+0000 7f0463eba640 1 Processor -- start 2024-04-15T15:20:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.445+0000 7f0463eba640 1 -- start start 2024-04-15T15:20:48.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0463eba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c15e6f0 0x7f045c06c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0463eba640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f045c06cf30 0x7f045c06e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0463eba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f045c06eae0 0x7f045c0ad840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0463eba640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f045c06d390 con 0x7f045c06cf30 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0463eba640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f045c06f220 con 0x7f045c06eae0 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0463eba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f045c06f500 con 0x7f045c15e6f0 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0461c56640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c15e6f0 0x7f045c06c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0462457640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f045c06eae0 0x7f045c0ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0462457640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f045c06eae0 0x7f045c0ad840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34752/0 (socket says 172.21.15.55:34752) 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0462457640 1 -- 172.21.15.55:0/398795682 learned_addr learned my addr 172.21.15.55:0/398795682 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0461c56640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c15e6f0 0x7f045c06c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.55:40462/0 (socket says 172.21.15.55:40462) 2024-04-15T15:20:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0461455640 1 --2- 172.21.15.55:0/398795682 >> v2:172.21.15.12:3300/0 conn(0x7f045c06cf30 0x7f045c06e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0462457640 1 -- 172.21.15.55:0/398795682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c15e6f0 msgr2=0x7f045c06c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0462457640 1 --2- 172.21.15.55:0/398795682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c15e6f0 0x7f045c06c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0462457640 1 -- 172.21.15.55:0/398795682 >> v2:172.21.15.12:3300/0 conn(0x7f045c06cf30 msgr2=0x7f045c06e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0462457640 1 --2- 172.21.15.55:0/398795682 >> v2:172.21.15.12:3300/0 conn(0x7f045c06cf30 0x7f045c06e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0462457640 1 -- 172.21.15.55:0/398795682 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c0b45d0 con 0x7f045c06eae0 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0461c56640 1 --2- 172.21.15.55:0/398795682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c15e6f0 0x7f045c06c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.446+0000 7f0461455640 1 --2- 172.21.15.55:0/398795682 >> v2:172.21.15.12:3300/0 conn(0x7f045c06cf30 0x7f045c06e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.447+0000 7f0462457640 1 --2- 172.21.15.55:0/398795682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06eae0 0x7f045c0ad840 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f045c0b38c0 tx=0x7f045402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eecf4349175c5a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.447+0000 7f0460c54640 1 -- 172.21.15.55:0/398795682 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045402ca70 con 0x7f045c06eae0 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.447+0000 7f0460c54640 1 -- 172.21.15.55:0/398795682 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f045402cc10 con 0x7f045c06eae0 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.447+0000 7f0460c54640 1 -- 172.21.15.55:0/398795682 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04540343f0 con 0x7f045c06eae0 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 -- 172.21.15.55:0/398795682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06eae0 msgr2=0x7f045c0ad840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 --2- 172.21.15.55:0/398795682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06eae0 0x7f045c0ad840 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f045c0b38c0 tx=0x7f045402bed0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 -- 172.21.15.55:0/398795682 shutdown_connections 2024-04-15T15:20:48.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 --2- 172.21.15.55:0/398795682 >> v2:172.21.15.12:3300/0 conn(0x7f045c06cf30 0x7f045c06e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 --2- 172.21.15.55:0/398795682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06eae0 0x7f045c0ad840 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 --2- 172.21.15.55:0/398795682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c15e6f0 0x7f045c06c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 -- 172.21.15.55:0/398795682 >> 172.21.15.55:0/398795682 conn(0x7f045c072440 msgr2=0x7f045c072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:48.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 -- 172.21.15.55:0/398795682 shutdown_connections 2024-04-15T15:20:48.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.448+0000 7f0463eba640 1 -- 172.21.15.55:0/398795682 wait complete. 2024-04-15T15:20:48.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.449+0000 7f0463eba640 1 Processor -- start 2024-04-15T15:20:48.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.449+0000 7f0463eba640 1 -- start start 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0463eba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06cf30 0x7f045c1f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0463eba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c06eae0 0x7f045c1f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0463eba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045c15e6f0 0x7f045c0b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0463eba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f045c0b41f0 con 0x7f045c15e6f0 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0463eba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f045c06ef30 con 0x7f045c06cf30 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0463eba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f045c0b4370 con 0x7f045c06eae0 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461c56640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06cf30 0x7f045c1f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461455640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c06eae0 0x7f045c1f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461c56640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06cf30 0x7f045c1f38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34762/0 (socket says 172.21.15.55:34762) 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461455640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c06eae0 0x7f045c1f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.55:40476/0 (socket says 172.21.15.55:40476) 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461c56640 1 -- 172.21.15.55:0/2596572950 learned_addr learned my addr 172.21.15.55:0/2596572950 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0462457640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045c15e6f0 0x7f045c0b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461455640 1 -- 172.21.15.55:0/2596572950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06cf30 msgr2=0x7f045c1f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461455640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06cf30 0x7f045c1f38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461455640 1 -- 172.21.15.55:0/2596572950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045c15e6f0 msgr2=0x7f045c0b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461455640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045c15e6f0 0x7f045c0b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461455640 1 -- 172.21.15.55:0/2596572950 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c1f7960 con 0x7f045c06eae0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461c56640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06cf30 0x7f045c1f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f0461455640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c06eae0 0x7f045c1f3e10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f044c00e7e0 tx=0x7f044c00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f04527fc640 1 -- 172.21.15.55:0/2596572950 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c00f040 con 0x7f045c06eae0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.450+0000 7f04527fc640 1 -- 172.21.15.55:0/2596572950 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044c004030 con 0x7f045c06eae0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.451+0000 7f04527fc640 1 -- 172.21.15.55:0/2596572950 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c004420 con 0x7f045c06eae0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.451+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045c1f7c40 con 0x7f045c06eae0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.451+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f045c143330 con 0x7f045c06eae0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.451+0000 7f04527fc640 1 -- 172.21.15.55:0/2596572950 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f044c00ca60 con 0x7f045c06eae0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.451+0000 7f04527fc640 1 -- 172.21.15.55:0/2596572950 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f044c013ac0 con 0x7f045c06eae0 2024-04-15T15:20:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.451+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f045c06cd20 con 0x7f045c06eae0 2024-04-15T15:20:48.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.453+0000 7f04527fc640 1 -- 172.21.15.55:0/2596572950 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f045c06cd20 con 0x7f045c06eae0 2024-04-15T15:20:48.465 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:48.462+0000 7fb2824da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T15:20:48.473 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:48.472+0000 7fc6f7041040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T15:20:48.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.527+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "osd new", "uuid": "36106542-b6bf-439a-a645-5426a655af97", "id": 7} v 0) -- 0x7f045c0690f0 con 0x7f045c06eae0 2024-04-15T15:20:48.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.531+0000 7f04527fc640 1 -- 172.21.15.55:0/2596572950 <== mon.2 v2:172.21.15.55:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36106542-b6bf-439a-a645-5426a655af97", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f044c017020 con 0x7f045c06eae0 2024-04-15T15:20:48.533 INFO:teuthology.orchestra.run.smithi055.stdout:7 2024-04-15T15:20:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c06eae0 msgr2=0x7f045c1f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c06eae0 0x7f045c1f3e10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f044c00e7e0 tx=0x7f044c00eca0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 shutdown_connections 2024-04-15T15:20:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f045c15e6f0 0x7f045c0b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f045c06eae0 0x7f045c1f3e10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 --2- 172.21.15.55:0/2596572950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06cf30 0x7f045c1f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 >> 172.21.15.55:0/2596572950 conn(0x7f045c072440 msgr2=0x7f045c1536a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 shutdown_connections 2024-04-15T15:20:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:20:48.532+0000 7f0463eba640 1 -- 172.21.15.55:0/2596572950 wait complete. 2024-04-15T15:20:48.544 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-15T15:20:48.544 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-15T15:20:48.546 INFO:tasks.ceph.osd.0:Started 2024-04-15T15:20:48.547 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-15T15:20:48.547 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-15T15:20:48.549 INFO:tasks.ceph.osd.1:Started 2024-04-15T15:20:48.549 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-15T15:20:48.549 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-15T15:20:48.551 INFO:tasks.ceph.osd.2:Started 2024-04-15T15:20:48.551 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-15T15:20:48.551 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-15T15:20:48.553 INFO:tasks.ceph.osd.3:Started 2024-04-15T15:20:48.553 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-15T15:20:48.554 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-15T15:20:48.556 INFO:tasks.ceph.osd.4:Started 2024-04-15T15:20:48.556 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-15T15:20:48.556 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-15T15:20:48.558 INFO:tasks.ceph.osd.5:Started 2024-04-15T15:20:48.558 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-15T15:20:48.558 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-15T15:20:48.560 INFO:tasks.ceph.osd.6:Started 2024-04-15T15:20:48.560 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-15T15:20:48.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-15T15:20:48.564 INFO:tasks.ceph.osd.7:Started 2024-04-15T15:20:48.565 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:20:48.605 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:48.603+0000 7fc6f7041040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T15:20:48.621 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:48.619+0000 7fb2824da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T15:20:48.668 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:48.667+0000 7fc6f7041040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T15:20:48.678 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:48.676+0000 7fb2824da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.692+0000 7f7e6d70e640 1 Processor -- start 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.692+0000 7f7e6d70e640 1 -- start start 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e6d70e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6815ed00 0x7f7e68070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e6d70e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7e680cf600 0x7f7e680cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e6d70e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e6806ef70 0x7f7e6806f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e6d70e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7e680cffa0 con 0x7f7e6806ef70 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e6d70e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7e680ae440 con 0x7f7e680cf600 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e6d70e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7e680ae720 con 0x7f7e6815ed00 2024-04-15T15:20:48.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e677fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e6806ef70 0x7f7e6806f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e5ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7e680cf600 0x7f7e680cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6815ed00 0x7f7e68070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.693+0000 7f7e677fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e6806ef70 0x7f7e6806f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60046/0 (socket says 172.21.15.12:60046) 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.694+0000 7f7e677fe640 1 -- 172.21.15.12:0/1457645774 learned_addr learned my addr 172.21.15.12:0/1457645774 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.694+0000 7f7e677fe640 1 -- 172.21.15.12:0/1457645774 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6815ed00 msgr2=0x7f7e68070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.694+0000 7f7e677fe640 1 --2- 172.21.15.12:0/1457645774 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6815ed00 0x7f7e68070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.694+0000 7f7e677fe640 1 -- 172.21.15.12:0/1457645774 >> v2:172.21.15.55:3300/0 conn(0x7f7e680cf600 msgr2=0x7f7e680cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.694+0000 7f7e677fe640 1 --2- 172.21.15.12:0/1457645774 >> v2:172.21.15.55:3300/0 conn(0x7f7e680cf600 0x7f7e680cfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.694+0000 7f7e677fe640 1 -- 172.21.15.12:0/1457645774 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e680aea00 con 0x7f7e6806ef70 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.694+0000 7f7e66ffd640 1 --2- 172.21.15.12:0/1457645774 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6815ed00 0x7f7e68070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.694+0000 7f7e677fe640 1 --2- 172.21.15.12:0/1457645774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6806ef70 0x7f7e6806f400 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e68071940 tx=0x7f7e6002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaeebddb942cf079 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:48.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e667fc640 1 -- 172.21.15.12:0/1457645774 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6002ca70 con 0x7f7e6806ef70 2024-04-15T15:20:48.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e667fc640 1 -- 172.21.15.12:0/1457645774 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6002cc10 con 0x7f7e6806ef70 2024-04-15T15:20:48.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e667fc640 1 -- 172.21.15.12:0/1457645774 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e600343f0 con 0x7f7e6806ef70 2024-04-15T15:20:48.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e6d70e640 1 -- 172.21.15.12:0/1457645774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6806ef70 msgr2=0x7f7e6806f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e6d70e640 1 --2- 172.21.15.12:0/1457645774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6806ef70 0x7f7e6806f400 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e68071940 tx=0x7f7e6002bed0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e6d70e640 1 -- 172.21.15.12:0/1457645774 shutdown_connections 2024-04-15T15:20:48.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e6d70e640 1 --2- 172.21.15.12:0/1457645774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e6806ef70 0x7f7e6806f400 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e6d70e640 1 --2- 172.21.15.12:0/1457645774 >> v2:172.21.15.55:3300/0 conn(0x7f7e680cf600 0x7f7e680cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e6d70e640 1 --2- 172.21.15.12:0/1457645774 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6815ed00 0x7f7e68070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e6d70e640 1 -- 172.21.15.12:0/1457645774 >> 172.21.15.12:0/1457645774 conn(0x7f7e6809a9d0 msgr2=0x7f7e680d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:48.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.695+0000 7f7e6d70e640 1 -- 172.21.15.12:0/1457645774 shutdown_connections 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.696+0000 7f7e6d70e640 1 -- 172.21.15.12:0/1457645774 wait complete. 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.697+0000 7f7e6d70e640 1 Processor -- start 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.697+0000 7f7e6d70e640 1 -- start start 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.697+0000 7f7e6d70e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6806ef70 0x7f7e680c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.697+0000 7f7e6d70e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e680cf600 0x7f7e680af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.697+0000 7f7e6d70e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e6815ed00 0x7f7e68153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.697+0000 7f7e6d70e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7e680701a0 con 0x7f7e680cf600 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e6d70e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7e680aefc0 con 0x7f7e6815ed00 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e6d70e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7e68070e60 con 0x7f7e6806ef70 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e680cf600 0x7f7e680af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e680cf600 0x7f7e680af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60056/0 (socket says 172.21.15.12:60056) 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 -- 172.21.15.12:0/2953689298 learned_addr learned my addr 172.21.15.12:0/2953689298 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:20:48.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e66ffd640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6806ef70 0x7f7e680c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e677fe640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e6815ed00 0x7f7e68153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 -- 172.21.15.12:0/2953689298 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6806ef70 msgr2=0x7f7e680c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6806ef70 0x7f7e680c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 -- 172.21.15.12:0/2953689298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e6815ed00 msgr2=0x7f7e68153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e6815ed00 0x7f7e68153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 -- 172.21.15.12:0/2953689298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e680c5bf0 con 0x7f7e680cf600 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5ffff640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e680cf600 0x7f7e680af750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000e8c0 tx=0x7f7e5000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5f7fe640 1 -- 172.21.15.12:0/2953689298 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50017070 con 0x7f7e680cf600 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5f7fe640 1 -- 172.21.15.12:0/2953689298 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e5000c940 con 0x7f7e680cf600 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e5f7fe640 1 -- 172.21.15.12:0/2953689298 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000cd10 con 0x7f7e680cf600 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e680c5e90 con 0x7f7e680cf600 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.698+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e680c98c0 con 0x7f7e680cf600 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.699+0000 7f7e5f7fe640 1 -- 172.21.15.12:0/2953689298 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7e5001a070 con 0x7f7e680cf600 2024-04-15T15:20:48.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.699+0000 7f7e5f7fe640 1 -- 172.21.15.12:0/2953689298 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f7e50006020 con 0x7f7e680cf600 2024-04-15T15:20:48.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.699+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e68068250 con 0x7f7e680cf600 2024-04-15T15:20:48.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.702+0000 7f7e5f7fe640 1 -- 172.21.15.12:0/2953689298 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7e68068250 con 0x7f7e680cf600 2024-04-15T15:20:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.764+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e680690f0 con 0x7f7e680cf600 2024-04-15T15:20:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.764+0000 7f7e5f7fe640 1 -- 172.21.15.12:0/2953689298 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f7e50006520 con 0x7f7e680cf600 2024-04-15T15:20:48.767 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:20:48.767 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:20:48.528878+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:20:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e680cf600 msgr2=0x7f7e680af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e680cf600 0x7f7e680af750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000e8c0 tx=0x7f7e5000ed80 comp rx=0 tx=0).stop 2024-04-15T15:20:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 shutdown_connections 2024-04-15T15:20:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e6815ed00 0x7f7e68153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e680cf600 0x7f7e680af750 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 --2- 172.21.15.12:0/2953689298 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7e6806ef70 0x7f7e680c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:48.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 >> 172.21.15.12:0/2953689298 conn(0x7f7e6809a9d0 msgr2=0x7f7e680b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:48.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 shutdown_connections 2024-04-15T15:20:48.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:20:48.765+0000 7f7e6d70e640 1 -- 172.21.15.12:0/2953689298 wait complete. 2024-04-15T15:20:48.778 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-15T15:20:48.779 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-15T15:20:48.786 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:48.785+0000 7fc6f7041040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:20:48.800 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:48.798+0000 7fb2824da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:20:48.885 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:48.883+0000 7fc6f7041040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T15:20:48.900 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:48.898+0000 7fb2824da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T15:20:49.047 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:49.045+0000 7fc6f7041040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:20:49.062 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:49.060+0000 7fb2824da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:20:49.090 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:49.089+0000 7fc6f7041040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T15:20:49.107 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:49.105+0000 7fb2824da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T15:20:49.152 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-04-15T15:20:49.150+0000 7f9d15414640 -1 Falling back to public interface 2024-04-15T15:20:49.157 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-04-15T15:20:49.154+0000 7f5e01d73640 -1 Falling back to public interface 2024-04-15T15:20:49.160 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-15T15:20:49.158+0000 7f0f5908b640 -1 Falling back to public interface 2024-04-15T15:20:49.167 INFO:tasks.ceph.osd.7.smithi055.stderr:2024-04-15T15:20:49.165+0000 7f359ec19640 -1 Falling back to public interface 2024-04-15T15:20:49.168 INFO:tasks.ceph.osd.4.smithi055.stderr:2024-04-15T15:20:49.166+0000 7f0a28e0b640 -1 Falling back to public interface 2024-04-15T15:20:49.184 INFO:tasks.ceph.osd.6.smithi055.stderr:2024-04-15T15:20:49.182+0000 7f9e54e0e640 -1 Falling back to public interface 2024-04-15T15:20:49.184 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-04-15T15:20:49.182+0000 7f8322215640 -1 Falling back to public interface 2024-04-15T15:20:49.193 INFO:tasks.ceph.osd.5.smithi055.stderr:2024-04-15T15:20:49.191+0000 7f9cd160f640 -1 Falling back to public interface 2024-04-15T15:20:49.258 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:49.257+0000 7fc6f7041040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T15:20:49.275 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:49.272+0000 7fb2824da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T15:20:49.584 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:20:49.677 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.675+0000 7f7879582640 1 Processor -- start 2024-04-15T15:20:49.677 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.675+0000 7f7879582640 1 -- start start 2024-04-15T15:20:49.679 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f7879582640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f78740b5620 0x7f78740b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:49.679 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f7879582640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f78740d0fe0 0x7f78740d1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:49.679 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f7879582640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78740cfed0 con 0x7f7874161e10 2024-04-15T15:20:49.679 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f7879582640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f78740d1970 con 0x7f78740b5620 2024-04-15T15:20:49.679 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f7879582640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f78740d23f0 con 0x7f78740d0fe0 2024-04-15T15:20:49.679 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f786bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f78740b5620 0x7f78740b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:49.679 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f786bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f78740b5620 0x7f78740b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60072/0 (socket says 172.21.15.12:60072) 2024-04-15T15:20:49.679 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f786bfff640 1 -- 172.21.15.12:0/3700497882 learned_addr learned my addr 172.21.15.12:0/3700497882 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:20:49.680 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f78737fe640 1 --2- 172.21.15.12:0/3700497882 >> v2:172.21.15.55:3300/0 conn(0x7f78740d0fe0 0x7f78740d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:49.680 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f786bfff640 1 -- 172.21.15.12:0/3700497882 >> v2:172.21.15.55:3300/0 conn(0x7f78740d0fe0 msgr2=0x7f78740d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:49.680 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f786bfff640 1 --2- 172.21.15.12:0/3700497882 >> v2:172.21.15.55:3300/0 conn(0x7f78740d0fe0 0x7f78740d1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:49.680 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f786bfff640 1 -- 172.21.15.12:0/3700497882 >> v1:172.21.15.55:6789/0 conn(0x7f7874161e10 legacy=0x7f78740d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:49.680 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.676+0000 7f786bfff640 1 -- 172.21.15.12:0/3700497882 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78740d01b0 con 0x7f78740b5620 2024-04-15T15:20:49.680 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.677+0000 7f786bfff640 1 --2- 172.21.15.12:0/3700497882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740b7460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f78740b5f70 tx=0x7f785c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00c332521b57977 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:49.681 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.677+0000 7f78727fc640 1 -- 172.21.15.12:0/3700497882 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c035020 con 0x7f78740b5620 2024-04-15T15:20:49.681 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.677+0000 7f78727fc640 1 -- 172.21.15.12:0/3700497882 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785c031870 con 0x7f78740b5620 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.677+0000 7f78727fc640 1 -- 172.21.15.12:0/3700497882 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c031c40 con 0x7f78740b5620 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.678+0000 7f7879582640 1 -- 172.21.15.12:0/3700497882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 msgr2=0x7f78740b7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.678+0000 7f7879582640 1 --2- 172.21.15.12:0/3700497882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740b7460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f78740b5f70 tx=0x7f785c02b910 comp rx=0 tx=0).stop 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.678+0000 7f7879582640 1 -- 172.21.15.12:0/3700497882 shutdown_connections 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.678+0000 7f7879582640 1 --2- 172.21.15.12:0/3700497882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740b7460 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.678+0000 7f7879582640 1 --2- 172.21.15.12:0/3700497882 >> v2:172.21.15.55:3300/0 conn(0x7f78740d0fe0 0x7f78740d1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.678+0000 7f7879582640 1 -- 172.21.15.12:0/3700497882 >> 172.21.15.12:0/3700497882 conn(0x7f787408bfd0 msgr2=0x7f78740d4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.678+0000 7f7879582640 1 -- 172.21.15.12:0/3700497882 shutdown_connections 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.678+0000 7f7879582640 1 -- 172.21.15.12:0/3700497882 wait complete. 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.679+0000 7f7879582640 1 Processor -- start 2024-04-15T15:20:49.682 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.679+0000 7f7879582640 1 -- start start 2024-04-15T15:20:49.684 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7879582640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:49.684 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7879582640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78740d0fe0 0x7f78740ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7879582640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7874161e10 0x7f78740b3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7879582640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7874162370 con 0x7f78740b5620 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7879582640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f78741621f0 con 0x7f78740d0fe0 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7879582640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7874162620 con 0x7f7874161e10 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60076/0 (socket says 172.21.15.12:60076) 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 -- 172.21.15.12:0/260727694 learned_addr learned my addr 172.21.15.12:0/260727694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f786bfff640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78740d0fe0 0x7f78740ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f78737fe640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7874161e10 0x7f78740b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 -- 172.21.15.12:0/260727694 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7874161e10 msgr2=0x7f78740b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7874161e10 0x7f78740b3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:49.685 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 -- 172.21.15.12:0/260727694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78740d0fe0 msgr2=0x7f78740ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78740d0fe0 0x7f78740ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 -- 172.21.15.12:0/260727694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78740c9f70 con 0x7f78740b5620 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.680+0000 7f7872ffd640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740c9a30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f786400c3c0 tx=0x7f786400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.681+0000 7f786b7fe640 1 -- 172.21.15.12:0/260727694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78640052d0 con 0x7f78740b5620 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.681+0000 7f786b7fe640 1 -- 172.21.15.12:0/260727694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7864005490 con 0x7f78740b5620 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.681+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78740b1c90 con 0x7f78740b5620 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.681+0000 7f786b7fe640 1 -- 172.21.15.12:0/260727694 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864007260 con 0x7f78740b5620 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.681+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78740b2250 con 0x7f78740b5620 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.681+0000 7f786b7fe640 1 -- 172.21.15.12:0/260727694 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7864007400 con 0x7f78740b5620 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.681+0000 7f786b7fe640 1 -- 172.21.15.12:0/260727694 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f7864011c60 con 0x7f78740b5620 2024-04-15T15:20:49.686 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.682+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7874075b20 con 0x7f78740b5620 2024-04-15T15:20:49.687 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.684+0000 7f786b7fe640 1 -- 172.21.15.12:0/260727694 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f78640035f0 con 0x7f78740b5620 2024-04-15T15:20:49.749 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.747+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78740d1d00 con 0x7f78740b5620 2024-04-15T15:20:49.750 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.747+0000 7f786b7fe640 1 -- 172.21.15.12:0/260727694 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f7864003a00 con 0x7f78740b5620 2024-04-15T15:20:49.750 INFO:teuthology.misc.health.smithi012.stdout: 2024-04-15T15:20:49.750 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":10,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:20:48.528878+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:20:49.751 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 msgr2=0x7f78740c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:49.751 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740c9a30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f786400c3c0 tx=0x7f786400c880 comp rx=0 tx=0).stop 2024-04-15T15:20:49.751 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 shutdown_connections 2024-04-15T15:20:49.751 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7874161e10 0x7f78740b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:49.751 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78740d0fe0 0x7f78740ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:49.752 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 --2- 172.21.15.12:0/260727694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78740b5620 0x7f78740c9a30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:49.752 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 >> 172.21.15.12:0/260727694 conn(0x7f787408bfd0 msgr2=0x7f78740b8040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:49.752 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 shutdown_connections 2024-04-15T15:20:49.752 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:49.748+0000 7f7879582640 1 -- 172.21.15.12:0/260727694 wait complete. 2024-04-15T15:20:49.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-15T15:20:50.114 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:50.112+0000 7fc6f7041040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T15:20:50.130 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:50.128+0000 7fb2824da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T15:20:50.404 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:50.402+0000 7fc6f7041040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T15:20:50.421 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:50.419+0000 7fb2824da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T15:20:50.456 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:50.455+0000 7fc6f7041040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T15:20:50.475 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:50.473+0000 7fb2824da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T15:20:50.577 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:50.576+0000 7fc6f7041040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T15:20:50.597 INFO:tasks.ceph.mgr.y.smithi012.stderr:2024-04-15T15:20:50.595+0000 7fb2824da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T15:20:51.593 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:51.592+0000 7fc697d51640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-15T15:20:51.595 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:20:51.594+0000 7fc697d51640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-15T15:20:52.633 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-04-15T15:20:52.631+0000 7f5e01d73640 -1 osd.2 0 log_to_monitors true 2024-04-15T15:20:52.651 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-04-15T15:20:52.649+0000 7f9d15414640 -1 osd.0 0 log_to_monitors true 2024-04-15T15:20:52.688 INFO:tasks.ceph.osd.7.smithi055.stderr:2024-04-15T15:20:52.686+0000 7f359ec19640 -1 osd.7 0 log_to_monitors true 2024-04-15T15:20:52.688 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-04-15T15:20:52.686+0000 7f8322215640 -1 osd.3 0 log_to_monitors true 2024-04-15T15:20:52.688 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-15T15:20:52.686+0000 7f0f5908b640 -1 osd.1 0 log_to_monitors true 2024-04-15T15:20:52.721 INFO:tasks.ceph.osd.6.smithi055.stderr:2024-04-15T15:20:52.719+0000 7f9e54e0e640 -1 osd.6 0 log_to_monitors true 2024-04-15T15:20:52.721 INFO:tasks.ceph.osd.4.smithi055.stderr:2024-04-15T15:20:52.719+0000 7f0a28e0b640 -1 osd.4 0 log_to_monitors true 2024-04-15T15:20:52.723 INFO:tasks.ceph.osd.5.smithi055.stderr:2024-04-15T15:20:52.722+0000 7f9cd160f640 -1 osd.5 0 log_to_monitors true 2024-04-15T15:20:56.497 INFO:tasks.ceph.osd.6.smithi055.stderr:2024-04-15T15:20:56.493+0000 7f9e51089640 -1 osd.6 0 waiting for initial osdmap 2024-04-15T15:20:56.506 INFO:tasks.ceph.osd.7.smithi055.stderr:2024-04-15T15:20:56.505+0000 7f359a5ec640 -1 osd.7 0 waiting for initial osdmap 2024-04-15T15:20:56.509 INFO:tasks.ceph.osd.6.smithi055.stderr:2024-04-15T15:20:56.508+0000 7f9e48695640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:20:56.517 INFO:tasks.ceph.osd.7.smithi055.stderr:2024-04-15T15:20:56.516+0000 7f359240c640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:20:56.523 INFO:tasks.ceph.osd.5.smithi055.stderr:2024-04-15T15:20:56.522+0000 7f9cccf7d640 -1 osd.5 0 waiting for initial osdmap 2024-04-15T15:20:56.530 INFO:tasks.ceph.osd.4.smithi055.stderr:2024-04-15T15:20:56.528+0000 7f0a24da1640 -1 osd.4 0 waiting for initial osdmap 2024-04-15T15:20:56.537 INFO:tasks.ceph.osd.5.smithi055.stderr:2024-04-15T15:20:56.536+0000 7f9cc4d9d640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:20:56.541 INFO:tasks.ceph.osd.4.smithi055.stderr:2024-04-15T15:20:56.540+0000 7f0a1cbc1640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:20:56.567 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:20:56.638 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-04-15T15:20:56.635+0000 7f9d10d61640 -1 osd.0 0 waiting for initial osdmap 2024-04-15T15:20:56.658 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-04-15T15:20:56.656+0000 7f9d08b81640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:20:56.711 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-04-15T15:20:56.709+0000 7f5dfdd09640 -1 osd.2 0 waiting for initial osdmap 2024-04-15T15:20:56.718 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.715+0000 7f446318d640 1 Processor -- start 2024-04-15T15:20:56.718 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.716+0000 7f446318d640 1 -- start start 2024-04-15T15:20:56.720 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.717+0000 7f446318d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f445c15c6f0 0x7f445c0ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:56.720 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.717+0000 7f446318d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f445c0cd460 0x7f445c0cd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:56.720 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.717+0000 7f446318d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f445c09b830 con 0x7f445c0d0090 2024-04-15T15:20:56.720 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.717+0000 7f446318d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f445c09bdf0 con 0x7f445c0cd460 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.717+0000 7f4460f29640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f445c15c6f0 0x7f445c0ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.717+0000 7f4460f29640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f445c15c6f0 0x7f445c0ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:45752/0 (socket says 172.21.15.12:45752) 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.717+0000 7f4460f29640 1 -- 172.21.15.12:0/1204956949 learned_addr learned my addr 172.21.15.12:0/1204956949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.718+0000 7f445bfff640 1 --2- 172.21.15.12:0/1204956949 >> v2:172.21.15.12:3300/0 conn(0x7f445c0cd460 0x7f445c0cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.718+0000 7f446318d640 1 -- 172.21.15.12:0/1204956949 --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f445c0cde50 con 0x7f445c15c6f0 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.718+0000 7f445b7fe640 1 -- 172.21.15.12:0/1204956949 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855763681 0 0) 0x7f445c09b830 con 0x7f445c0d0090 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.718+0000 7f445bfff640 1 -- 172.21.15.12:0/1204956949 >> v2:172.21.15.55:3300/0 conn(0x7f445c15c6f0 msgr2=0x7f445c0ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.718+0000 7f445bfff640 1 --2- 172.21.15.12:0/1204956949 >> v2:172.21.15.55:3300/0 conn(0x7f445c15c6f0 0x7f445c0ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.718+0000 7f445bfff640 1 -- 172.21.15.12:0/1204956949 >> v1:172.21.15.55:6789/0 conn(0x7f445c0d0090 legacy=0x7f445c0d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:56.721 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.718+0000 7f445bfff640 1 -- 172.21.15.12:0/1204956949 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c09bb10 con 0x7f445c0cd460 2024-04-15T15:20:56.722 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.719+0000 7f445bfff640 1 --2- 172.21.15.12:0/1204956949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c0cd910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f445c06f6b0 tx=0x7f444802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b2eee9be0694 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:56.722 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.719+0000 7f445b7fe640 1 -- 172.21.15.12:0/1204956949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44480315a0 con 0x7f445c0cd460 2024-04-15T15:20:56.722 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.719+0000 7f445b7fe640 1 -- 172.21.15.12:0/1204956949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4448037070 con 0x7f445c0cd460 2024-04-15T15:20:56.722 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.719+0000 7f445b7fe640 1 -- 172.21.15.12:0/1204956949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44480315a0 con 0x7f445c0cd460 2024-04-15T15:20:56.723 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.720+0000 7f446318d640 1 -- 172.21.15.12:0/1204956949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 msgr2=0x7f445c0cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:56.723 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.720+0000 7f446318d640 1 --2- 172.21.15.12:0/1204956949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c0cd910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f445c06f6b0 tx=0x7f444802b840 comp rx=0 tx=0).stop 2024-04-15T15:20:56.723 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.721+0000 7f446318d640 1 -- 172.21.15.12:0/1204956949 shutdown_connections 2024-04-15T15:20:56.723 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.721+0000 7f446318d640 1 --2- 172.21.15.12:0/1204956949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c0cd910 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.724 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.721+0000 7f446318d640 1 --2- 172.21.15.12:0/1204956949 >> v2:172.21.15.55:3300/0 conn(0x7f445c15c6f0 0x7f445c0ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.724 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.721+0000 7f446318d640 1 -- 172.21.15.12:0/1204956949 >> 172.21.15.12:0/1204956949 conn(0x7f445c0b3f40 msgr2=0x7f445c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:56.724 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.721+0000 7f446318d640 1 -- 172.21.15.12:0/1204956949 shutdown_connections 2024-04-15T15:20:56.724 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.721+0000 7f446318d640 1 -- 172.21.15.12:0/1204956949 wait complete. 2024-04-15T15:20:56.724 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-04-15T15:20:56.722+0000 7f5df5b29640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:20:56.725 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.723+0000 7f446318d640 1 Processor -- start 2024-04-15T15:20:56.725 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.723+0000 7f446318d640 1 -- start start 2024-04-15T15:20:56.727 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f446318d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c15aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f4460f29640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c15aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f4460f29640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c15aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53394/0 (socket says 172.21.15.12:53394) 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f446318d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f445c0d0090 0x7f445c15e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f4460f29640 1 -- 172.21.15.12:0/218645425 learned_addr learned my addr 172.21.15.12:0/218645425 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f446318d640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f445c15c6f0 0x7f445c0c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f445c15cad0 con 0x7f445c0cd460 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f445c0b20a0 con 0x7f445c15c6f0 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.724+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f445c09b450 con 0x7f445c0d0090 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f446172a640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f445c15c6f0 0x7f445c0c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f4460f29640 1 -- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f445c0d0090 msgr2=0x7f445c15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f4460f29640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f445c0d0090 0x7f445c15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f4460f29640 1 -- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f445c15c6f0 msgr2=0x7f445c0c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:56.728 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f4460f29640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f445c15c6f0 0x7f445c0c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.729 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f4460f29640 1 -- 172.21.15.12:0/218645425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c1f4f80 con 0x7f445c0cd460 2024-04-15T15:20:56.729 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f446172a640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f445c15c6f0 0x7f445c0c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:20:56.729 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f4460f29640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c15aa00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4450009fd0 tx=0x7f445000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:56.729 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f44597fa640 1 -- 172.21.15.12:0/218645425 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4450017070 con 0x7f445c0cd460 2024-04-15T15:20:56.729 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f44597fa640 1 -- 172.21.15.12:0/218645425 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f445000c940 con 0x7f445c0cd460 2024-04-15T15:20:56.729 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445c1f5260 con 0x7f445c0cd460 2024-04-15T15:20:56.729 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.725+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f445c0b7c30 con 0x7f445c0cd460 2024-04-15T15:20:56.729 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.726+0000 7f44597fa640 1 -- 172.21.15.12:0/218645425 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445000cd10 con 0x7f445c0cd460 2024-04-15T15:20:56.730 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.728+0000 7f44597fa640 1 -- 172.21.15.12:0/218645425 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f445001a070 con 0x7f445c0cd460 2024-04-15T15:20:56.730 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.728+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f445c075b20 con 0x7f445c0cd460 2024-04-15T15:20:56.730 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.728+0000 7f44597fa640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f444006c860 0x7f444006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:20:56.731 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.728+0000 7f44597fa640 1 -- 172.21.15.12:0/218645425 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f445008ad00 con 0x7f445c0cd460 2024-04-15T15:20:56.731 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.728+0000 7f445bfff640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f444006c860 0x7f444006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:20:56.731 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-15T15:20:56.728+0000 7f0f55021640 -1 osd.1 0 waiting for initial osdmap 2024-04-15T15:20:56.731 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.729+0000 7f445bfff640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f444006c860 0x7f444006ed20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4448036000 tx=0x7f4448030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:20:56.738 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.736+0000 7f44597fa640 1 -- 172.21.15.12:0/218645425 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f44500600d0 con 0x7f445c0cd460 2024-04-15T15:20:56.749 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-04-15T15:20:56.747+0000 7f0f4ce41640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:20:56.758 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-04-15T15:20:56.756+0000 7f831dc7d640 -1 osd.3 0 waiting for initial osdmap 2024-04-15T15:20:56.769 INFO:tasks.ceph.osd.3.smithi012.stderr:2024-04-15T15:20:56.767+0000 7f8315a9d640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:20:56.934 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.932+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f445c0d1600 con 0x7f445c0cd460 2024-04-15T15:20:56.934 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.932+0000 7f44597fa640 1 -- 172.21.15.12:0/218645425 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f4450059f00 con 0x7f445c0cd460 2024-04-15T15:20:56.935 INFO:teuthology.misc.health.smithi012.stdout: 2024-04-15T15:20:56.935 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":12,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:20:54.596082+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:20:56.937 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f444006c860 msgr2=0x7f444006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:56.937 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f444006c860 0x7f444006ed20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4448036000 tx=0x7f4448030000 comp rx=0 tx=0).stop 2024-04-15T15:20:56.937 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 msgr2=0x7f445c15aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:20:56.937 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c15aa00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4450009fd0 tx=0x7f445000ed40 comp rx=0 tx=0).stop 2024-04-15T15:20:56.937 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 shutdown_connections 2024-04-15T15:20:56.937 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f444006c860 0x7f444006ed20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.937 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f445c15c6f0 0x7f445c0c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.938 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f445c0d0090 0x7f445c15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.938 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 --2- 172.21.15.12:0/218645425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f445c0cd460 0x7f445c15aa00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:20:56.938 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 >> 172.21.15.12:0/218645425 conn(0x7f445c0b3f40 msgr2=0x7f445c06e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:20:56.938 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 shutdown_connections 2024-04-15T15:20:56.938 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:20:56.934+0000 7f446318d640 1 -- 172.21.15.12:0/218645425 wait complete. 2024-04-15T15:20:56.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-15T15:21:01.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:21:01.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:21:03.753 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:03.857 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.855+0000 7f2653274640 1 Processor -- start 2024-04-15T15:21:03.858 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.855+0000 7f2653274640 1 -- start start 2024-04-15T15:21:03.859 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.856+0000 7f2653274640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f264c161e10 0x7f264c0b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:03.859 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.856+0000 7f2653274640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f264c0b5620 0x7f264c0d4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.856+0000 7f2653274640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f264c0cfed0 con 0x7f264c0d0fe0 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.856+0000 7f2653274640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f264c0d23f0 con 0x7f264c0b5620 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.856+0000 7f2653274640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f264c0d01b0 con 0x7f264c161e10 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.856+0000 7f265080f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f264c0b5620 0x7f264c0d4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.856+0000 7f265080f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f264c0b5620 0x7f264c0d4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35900/0 (socket says 172.21.15.12:35900) 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.856+0000 7f265080f640 1 -- 172.21.15.12:0/3180809576 learned_addr learned my addr 172.21.15.12:0/3180809576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.857+0000 7f2651010640 1 --2- 172.21.15.12:0/3180809576 >> v2:172.21.15.55:3300/0 conn(0x7f264c161e10 0x7f264c0b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.857+0000 7f265080f640 1 -- 172.21.15.12:0/3180809576 >> v2:172.21.15.55:3300/0 conn(0x7f264c161e10 msgr2=0x7f264c0b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.857+0000 7f265080f640 1 --2- 172.21.15.12:0/3180809576 >> v2:172.21.15.55:3300/0 conn(0x7f264c161e10 0x7f264c0b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.857+0000 7f265080f640 1 -- 172.21.15.12:0/3180809576 >> v1:172.21.15.55:6789/0 conn(0x7f264c0d0fe0 legacy=0x7f264c0d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:03.860 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.857+0000 7f265080f640 1 -- 172.21.15.12:0/3180809576 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f264c0d0490 con 0x7f264c0b5620 2024-04-15T15:21:03.861 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f265080f640 1 --2- 172.21.15.12:0/3180809576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c0d4f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f264c0d4570 tx=0x7f264002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3e5fa1c8fa4ac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:03.861 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f264bfff640 1 -- 172.21.15.12:0/3180809576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264002e070 con 0x7f264c0b5620 2024-04-15T15:21:03.861 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f264bfff640 1 -- 172.21.15.12:0/3180809576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2640037070 con 0x7f264c0b5620 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f264bfff640 1 -- 172.21.15.12:0/3180809576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640031a00 con 0x7f264c0b5620 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f2653274640 1 -- 172.21.15.12:0/3180809576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 msgr2=0x7f264c0d4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f2653274640 1 --2- 172.21.15.12:0/3180809576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c0d4f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f264c0d4570 tx=0x7f264002b880 comp rx=0 tx=0).stop 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f2653274640 1 -- 172.21.15.12:0/3180809576 shutdown_connections 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f2653274640 1 --2- 172.21.15.12:0/3180809576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c0d4f30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f2653274640 1 --2- 172.21.15.12:0/3180809576 >> v2:172.21.15.55:3300/0 conn(0x7f264c161e10 0x7f264c0b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.858+0000 7f2653274640 1 -- 172.21.15.12:0/3180809576 >> 172.21.15.12:0/3180809576 conn(0x7f264c08bfd0 msgr2=0x7f264c0d4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.859+0000 7f2653274640 1 -- 172.21.15.12:0/3180809576 shutdown_connections 2024-04-15T15:21:03.862 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.859+0000 7f2653274640 1 -- 172.21.15.12:0/3180809576 wait complete. 2024-04-15T15:21:03.865 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.860+0000 7f2653274640 1 Processor -- start 2024-04-15T15:21:03.865 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.860+0000 7f2653274640 1 -- start start 2024-04-15T15:21:03.865 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.860+0000 7f2653274640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c15ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:03.866 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.860+0000 7f2653274640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f264c0d0fe0 0x7f264c0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:03.866 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2653274640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f264c161e10 0x7f264c0cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:03.866 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2653274640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f264c162370 con 0x7f264c0b5620 2024-04-15T15:21:03.866 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2653274640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f264c1621f0 con 0x7f264c0d0fe0 2024-04-15T15:21:03.866 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2653274640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f264c0d1430 con 0x7f264c161e10 2024-04-15T15:21:03.866 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c15ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:03.866 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c15ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35904/0 (socket says 172.21.15.12:35904) 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 -- 172.21.15.12:0/3204272679 learned_addr learned my addr 172.21.15.12:0/3204272679 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f265080f640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f264c0d0fe0 0x7f264c0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651811640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f264c161e10 0x7f264c0cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 -- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f264c161e10 msgr2=0x7f264c0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f264c161e10 0x7f264c0cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 -- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f264c0d0fe0 msgr2=0x7f264c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f264c0d0fe0 0x7f264c0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 -- 172.21.15.12:0/3204272679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f264c15cf40 con 0x7f264c0b5620 2024-04-15T15:21:03.867 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f265080f640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f264c0d0fe0 0x7f264c0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:03.868 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2651010640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c15ca00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f264400d5f0 tx=0x7f264400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:03.868 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2649ffb640 1 -- 172.21.15.12:0/3204272679 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26440125f0 con 0x7f264c0b5620 2024-04-15T15:21:03.868 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2649ffb640 1 -- 172.21.15.12:0/3204272679 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2644019070 con 0x7f264c0b5620 2024-04-15T15:21:03.868 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f264c0c9d90 con 0x7f264c0b5620 2024-04-15T15:21:03.868 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.861+0000 7f2649ffb640 1 -- 172.21.15.12:0/3204272679 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26440129d0 con 0x7f264c0b5620 2024-04-15T15:21:03.868 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.862+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f264c0ca350 con 0x7f264c0b5620 2024-04-15T15:21:03.868 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.863+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f264c075b20 con 0x7f264c0b5620 2024-04-15T15:21:03.868 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.863+0000 7f2649ffb640 1 -- 172.21.15.12:0/3204272679 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2644012b70 con 0x7f264c0b5620 2024-04-15T15:21:03.869 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.863+0000 7f2649ffb640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f263806c8f0 0x7f263806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:03.870 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.864+0000 7f2649ffb640 1 -- 172.21.15.12:0/3204272679 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f264408bbc0 con 0x7f264c0b5620 2024-04-15T15:21:03.870 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.864+0000 7f265080f640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f263806c8f0 0x7f263806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:03.870 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.864+0000 7f265080f640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f263806c8f0 0x7f263806edb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2640036000 tx=0x7f2640008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:03.870 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:03.868+0000 7f2649ffb640 1 -- 172.21.15.12:0/3204272679 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f264405a340 con 0x7f264c0b5620 2024-04-15T15:21:04.059 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.057+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f264c0d1d00 con 0x7f264c0b5620 2024-04-15T15:21:04.060 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.058+0000 7f2649ffb640 1 -- 172.21.15.12:0/3204272679 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7f2644059c40 con 0x7f264c0b5620 2024-04-15T15:21:04.060 INFO:teuthology.misc.health.smithi012.stdout: 2024-04-15T15:21:04.060 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":17,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:01.523736+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:04.062 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f263806c8f0 msgr2=0x7f263806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.062 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f263806c8f0 0x7f263806edb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2640036000 tx=0x7f2640008030 comp rx=0 tx=0).stop 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 msgr2=0x7f264c15ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c15ca00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f264400d5f0 tx=0x7f264400dab0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 shutdown_connections 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f264c161e10 0x7f264c0cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f263806c8f0 0x7f263806edb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f264c0d0fe0 0x7f264c0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 --2- 172.21.15.12:0/3204272679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f264c0b5620 0x7f264c15ca00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 >> 172.21.15.12:0/3204272679 conn(0x7f264c08bfd0 msgr2=0x7f264c0d2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 shutdown_connections 2024-04-15T15:21:04.063 INFO:teuthology.misc.health.smithi012.stderr:2024-04-15T15:21:04.060+0000 7f2653274640 1 -- 172.21.15.12:0/3204272679 wait complete. 2024-04-15T15:21:04.072 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-15T15:21:04.072 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-15T15:21:04.072 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-15T15:21:04.072 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-15T15:21:04.075 INFO:tasks.ceph.mds.a:Started 2024-04-15T15:21:04.075 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-15T15:21:04.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-15T15:21:04.077 INFO:tasks.ceph.mds.c:Started 2024-04-15T15:21:04.077 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-15T15:21:04.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-15T15:21:04.081 INFO:tasks.ceph.mds.b:Started 2024-04-15T15:21:04.081 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-15T15:21:04.081 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-15T15:21:04.083 INFO:tasks.ceph.mds.d:Started 2024-04-15T15:21:04.083 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-15T15:21:04.084 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:04.154 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-04-15T15:21:04.154 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-04-15T15:21:04.171 INFO:tasks.ceph.mds.b.smithi055.stdout:starting mds.b at 2024-04-15T15:21:04.172 INFO:tasks.ceph.mds.d.smithi055.stdout:starting mds.d at 2024-04-15T15:21:04.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.190+0000 7f0a9afa8640 1 Processor -- start 2024-04-15T15:21:04.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.190+0000 7f0a9afa8640 1 -- start start 2024-04-15T15:21:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.191+0000 7f0a9afa8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a940c4c70 0x7f0a9406c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.191+0000 7f0a9afa8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940cde10 0x7f0a940cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.191+0000 7f0a9afa8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a940d1f40 con 0x7f0a94162cd0 2024-04-15T15:21:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.191+0000 7f0a9afa8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0a940ce360 con 0x7f0a940c4c70 2024-04-15T15:21:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.191+0000 7f0a9afa8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0a940ce640 con 0x7f0a940cde10 2024-04-15T15:21:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.191+0000 7f0a93fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a940c4c70 0x7f0a9406c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.191+0000 7f0a93fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a940c4c70 0x7f0a9406c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35928/0 (socket says 172.21.15.12:35928) 2024-04-15T15:21:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.191+0000 7f0a93fff640 1 -- 172.21.15.12:0/1523455440 learned_addr learned my addr 172.21.15.12:0/1523455440 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.192+0000 7f0a99545640 1 --2- 172.21.15.12:0/1523455440 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940cde10 0x7f0a940cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.192+0000 7f0a93fff640 1 -- 172.21.15.12:0/1523455440 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940cde10 msgr2=0x7f0a940cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.192+0000 7f0a93fff640 1 --2- 172.21.15.12:0/1523455440 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940cde10 0x7f0a940cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.192+0000 7f0a93fff640 1 -- 172.21.15.12:0/1523455440 >> v1:172.21.15.55:6789/0 conn(0x7f0a94162cd0 legacy=0x7f0a940c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.192+0000 7f0a93fff640 1 -- 172.21.15.12:0/1523455440 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a940ce920 con 0x7f0a940c4c70 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.192+0000 7f0a93fff640 1 --2- 172.21.15.12:0/1523455440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940c4c70 0x7f0a9406c3f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009870 tx=0x7f0a8802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b53a137f1ad0257 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a937fe640 1 -- 172.21.15.12:0/1523455440 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88034070 con 0x7f0a940c4c70 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a937fe640 1 -- 172.21.15.12:0/1523455440 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a88030840 con 0x7f0a940c4c70 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a937fe640 1 -- 172.21.15.12:0/1523455440 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88030c10 con 0x7f0a940c4c70 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a9afa8640 1 -- 172.21.15.12:0/1523455440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940c4c70 msgr2=0x7f0a9406c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/1523455440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940c4c70 0x7f0a9406c3f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009870 tx=0x7f0a8802b980 comp rx=0 tx=0).stop 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a9afa8640 1 -- 172.21.15.12:0/1523455440 shutdown_connections 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/1523455440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940c4c70 0x7f0a9406c3f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/1523455440 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940cde10 0x7f0a940cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a9afa8640 1 -- 172.21.15.12:0/1523455440 >> 172.21.15.12:0/1523455440 conn(0x7f0a9409a9d0 msgr2=0x7f0a940d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a9afa8640 1 -- 172.21.15.12:0/1523455440 shutdown_connections 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.193+0000 7f0a9afa8640 1 -- 172.21.15.12:0/1523455440 wait complete. 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.194+0000 7f0a9afa8640 1 Processor -- start 2024-04-15T15:21:04.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.194+0000 7f0a9afa8640 1 -- start start 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a9afa8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940c4c70 0x7f0a941f8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a9afa8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940cde10 0x7f0a940b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a9afa8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a94162cd0 0x7f0a94164360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a9afa8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0a941630b0 con 0x7f0a940cde10 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a9afa8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0a940ceee0 con 0x7f0a94162cd0 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a9afa8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0a940d03e0 con 0x7f0a940c4c70 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a93fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940cde10 0x7f0a940b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a93fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940cde10 0x7f0a940b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35938/0 (socket says 172.21.15.12:35938) 2024-04-15T15:21:04.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a93fff640 1 -- 172.21.15.12:0/3342716117 learned_addr learned my addr 172.21.15.12:0/3342716117 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a98d44640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940c4c70 0x7f0a941f8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.195+0000 7f0a99545640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a94162cd0 0x7f0a94164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a93fff640 1 -- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940c4c70 msgr2=0x7f0a941f8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a93fff640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940c4c70 0x7f0a941f8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a93fff640 1 -- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a94162cd0 msgr2=0x7f0a94164360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a93fff640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a94162cd0 0x7f0a94164360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a93fff640 1 -- 172.21.15.12:0/3342716117 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a940b2bf0 con 0x7f0a940cde10 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a93fff640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940cde10 0x7f0a940b05e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0a88000c00 tx=0x7f0a88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a917fa640 1 -- 172.21.15.12:0/3342716117 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88048070 con 0x7f0a940cde10 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a917fa640 1 -- 172.21.15.12:0/3342716117 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a88030640 con 0x7f0a940cde10 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a917fa640 1 -- 172.21.15.12:0/3342716117 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88034040 con 0x7f0a940cde10 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a940b2e90 con 0x7f0a940cde10 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.196+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a940b1190 con 0x7f0a940cde10 2024-04-15T15:21:04.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.197+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a940690f0 con 0x7f0a940cde10 2024-04-15T15:21:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.197+0000 7f0a917fa640 1 -- 172.21.15.12:0/3342716117 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a8804b070 con 0x7f0a940cde10 2024-04-15T15:21:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.198+0000 7f0a917fa640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c8f0 0x7f0a8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.198+0000 7f0a917fa640 1 -- 172.21.15.12:0/3342716117 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0a880ad720 con 0x7f0a940cde10 2024-04-15T15:21:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.198+0000 7f0a98d44640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c8f0 0x7f0a8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.199+0000 7f0a98d44640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c8f0 0x7f0a8006edb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a84007fd0 tx=0x7f0a84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.202+0000 7f0a917fa640 1 -- 172.21.15.12:0/3342716117 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a8807bea0 con 0x7f0a940cde10 2024-04-15T15:21:04.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.390+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a940c46e0 con 0x7f0a940cde10 2024-04-15T15:21:04.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.391+0000 7f0a917fa640 1 -- 172.21.15.12:0/3342716117 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7f0a8807b7a0 con 0x7f0a940cde10 2024-04-15T15:21:04.393 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:04.394 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:01.523736+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:04.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c8f0 msgr2=0x7f0a8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c8f0 0x7f0a8006edb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a84007fd0 tx=0x7f0a84009040 comp rx=0 tx=0).stop 2024-04-15T15:21:04.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940cde10 msgr2=0x7f0a940b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940cde10 0x7f0a940b05e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0a88000c00 tx=0x7f0a88004730 comp rx=0 tx=0).stop 2024-04-15T15:21:04.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 shutdown_connections 2024-04-15T15:21:04.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c8f0 0x7f0a8006edb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a94162cd0 0x7f0a94164360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a940cde10 0x7f0a940b05e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 --2- 172.21.15.12:0/3342716117 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a940c4c70 0x7f0a941f8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.393+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 >> 172.21.15.12:0/3342716117 conn(0x7f0a9409a9d0 msgr2=0x7f0a94160aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:04.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.394+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 shutdown_connections 2024-04-15T15:21:04.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.394+0000 7f0a9afa8640 1 -- 172.21.15.12:0/3342716117 wait complete. 2024-04-15T15:21:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:21:04.407 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:04.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.511+0000 7f7f139d4640 1 Processor -- start 2024-04-15T15:21:04.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.512+0000 7f7f139d4640 1 -- start start 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.512+0000 7f7f139d4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7f0c0d3240 0x7f7f0c0d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.512+0000 7f7f139d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d1440 0x7f7f0c0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.512+0000 7f7f139d4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f0c0d2480 con 0x7f7f0c15e790 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.512+0000 7f7f139d4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7f0c0d2db0 con 0x7f7f0c0d3240 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.512+0000 7f7f139d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0c0d2760 con 0x7f7f0c0d1440 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f11f71640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d1440 0x7f7f0c0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f10f6f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7f0c0d3240 0x7f7f0c0d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f11f71640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d1440 0x7f7f0c0b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:35524/0 (socket says 172.21.15.12:35524) 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f11770640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f7f0c15e790 0x7f7f0c0cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:45274/0 (socket says 172.21.15.12:45274) 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f11f71640 1 -- 172.21.15.12:0/901603199 learned_addr learned my addr 172.21.15.12:0/901603199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f11f71640 1 -- 172.21.15.12:0/901603199 >> v2:172.21.15.55:3300/0 conn(0x7f7f0c0d3240 msgr2=0x7f7f0c0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:04.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f11f71640 1 --2- 172.21.15.12:0/901603199 >> v2:172.21.15.55:3300/0 conn(0x7f7f0c0d3240 0x7f7f0c0d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f11f71640 1 -- 172.21.15.12:0/901603199 >> v1:172.21.15.55:6789/0 conn(0x7f7f0c15e790 legacy=0x7f7f0c0cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f11f71640 1 -- 172.21.15.12:0/901603199 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c0d2a40 con 0x7f7f0c0d1440 2024-04-15T15:21:04.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.513+0000 7f7f03fff640 1 -- 172.21.15.12:0/901603199 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1225388853 0 0) 0x7f7f0c0d2480 con 0x7f7f0c15e790 2024-04-15T15:21:04.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.514+0000 7f7f11f71640 1 --2- 172.21.15.12:0/901603199 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d1440 0x7f7f0c0b74f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009e50 tx=0x7f7f0402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67d9686dc549eaeb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.514+0000 7f7f03fff640 1 -- 172.21.15.12:0/901603199 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0402cb00 con 0x7f7f0c0d1440 2024-04-15T15:21:04.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.514+0000 7f7f03fff640 1 -- 172.21.15.12:0/901603199 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0402cca0 con 0x7f7f0c0d1440 2024-04-15T15:21:04.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.514+0000 7f7f03fff640 1 -- 172.21.15.12:0/901603199 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04034610 con 0x7f7f0c0d1440 2024-04-15T15:21:04.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.514+0000 7f7f139d4640 1 -- 172.21.15.12:0/901603199 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d1440 msgr2=0x7f7f0c0b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.514+0000 7f7f139d4640 1 --2- 172.21.15.12:0/901603199 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d1440 0x7f7f0c0b74f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009e50 tx=0x7f7f0402bf60 comp rx=0 tx=0).stop 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.515+0000 7f7f139d4640 1 -- 172.21.15.12:0/901603199 shutdown_connections 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.515+0000 7f7f139d4640 1 --2- 172.21.15.12:0/901603199 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d1440 0x7f7f0c0b74f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.515+0000 7f7f139d4640 1 --2- 172.21.15.12:0/901603199 >> v2:172.21.15.55:3300/0 conn(0x7f7f0c0d3240 0x7f7f0c0d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.515+0000 7f7f139d4640 1 -- 172.21.15.12:0/901603199 >> 172.21.15.12:0/901603199 conn(0x7f7f0c08bf60 msgr2=0x7f7f0c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.515+0000 7f7f139d4640 1 -- 172.21.15.12:0/901603199 shutdown_connections 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.515+0000 7f7f139d4640 1 -- 172.21.15.12:0/901603199 wait complete. 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.516+0000 7f7f139d4640 1 Processor -- start 2024-04-15T15:21:04.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.516+0000 7f7f139d4640 1 -- start start 2024-04-15T15:21:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f139d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c0d1440 0x7f7f0c14fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f139d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d3240 0x7f7f0c0b8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f139d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f0c15e790 0x7f7f0c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f139d4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c0d5180 con 0x7f7f0c0d1440 2024-04-15T15:21:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f139d4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c0d0430 con 0x7f7f0c15e790 2024-04-15T15:21:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f139d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0c0d5300 con 0x7f7f0c0d3240 2024-04-15T15:21:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c0d1440 0x7f7f0c14fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c0d1440 0x7f7f0c14fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35944/0 (socket says 172.21.15.12:35944) 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 -- 172.21.15.12:0/346296804 learned_addr learned my addr 172.21.15.12:0/346296804 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11f71640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f0c15e790 0x7f7f0c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f10f6f640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d3240 0x7f7f0c0b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 -- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d3240 msgr2=0x7f7f0c0b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d3240 0x7f7f0c0b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 -- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f0c15e790 msgr2=0x7f7f0c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f0c15e790 0x7f7f0c0b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 -- 172.21.15.12:0/346296804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c0c99e0 con 0x7f7f0c0d1440 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f10f6f640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d3240 0x7f7f0c0b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f11770640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c0d1440 0x7f7f0c14fbf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800c430 tx=0x7f7f0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f01ffb640 1 -- 172.21.15.12:0/346296804 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f080074e0 con 0x7f7f0c0d1440 2024-04-15T15:21:04.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f01ffb640 1 -- 172.21.15.12:0/346296804 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f080047e0 con 0x7f7f0c0d1440 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0c0c9cc0 con 0x7f7f0c0d1440 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.517+0000 7f7f01ffb640 1 -- 172.21.15.12:0/346296804 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08004bd0 con 0x7f7f0c0d1440 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.518+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0c09ff10 con 0x7f7f0c0d1440 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.518+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0c0689e0 con 0x7f7f0c0d1440 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.519+0000 7f7f01ffb640 1 -- 172.21.15.12:0/346296804 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7f08011070 con 0x7f7f0c0d1440 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.519+0000 7f7f01ffb640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ef006c8f0 0x7f7ef006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.519+0000 7f7f01ffb640 1 -- 172.21.15.12:0/346296804 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f7f0808b450 con 0x7f7f0c0d1440 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.520+0000 7f7f10f6f640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ef006c8f0 0x7f7ef006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.520+0000 7f7f10f6f640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ef006c8f0 0x7f7ef006edb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0a2480 tx=0x7f7efc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.524+0000 7f7f01ffb640 1 -- 172.21.15.12:0/346296804 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f08059c60 con 0x7f7f0c0d1440 2024-04-15T15:21:04.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.708+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7f0c0b8450 con 0x7f7f0c0d1440 2024-04-15T15:21:04.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.709+0000 7f7f01ffb640 1 -- 172.21.15.12:0/346296804 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f7f08059560 con 0x7f7f0c0d1440 2024-04-15T15:21:04.711 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:04.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ef006c8f0 msgr2=0x7f7ef006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ef006c8f0 0x7f7ef006edb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0a2480 tx=0x7f7efc008040 comp rx=0 tx=0).stop 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c0d1440 msgr2=0x7f7f0c14fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c0d1440 0x7f7f0c14fbf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800c430 tx=0x7f7f0800c8f0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 shutdown_connections 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ef006c8f0 0x7f7ef006edb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f0c15e790 0x7f7f0c0b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f0c0d3240 0x7f7f0c0b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 --2- 172.21.15.12:0/346296804 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c0d1440 0x7f7f0c14fbf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 >> 172.21.15.12:0/346296804 conn(0x7f7f0c08bf60 msgr2=0x7f7f0c0caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 shutdown_connections 2024-04-15T15:21:04.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.711+0000 7f7f139d4640 1 -- 172.21.15.12:0/346296804 wait complete. 2024-04-15T15:21:04.724 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:21:04.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.825+0000 7fa0b3b1c640 1 Processor -- start 2024-04-15T15:21:04.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.825+0000 7fa0b3b1c640 1 -- start start 2024-04-15T15:21:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b3b1c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0ac0d2360 0x7fa0ac0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b3b1c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 0x7fa0ac0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b3b1c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ac0b47a0 con 0x7fa0ac15e700 2024-04-15T15:21:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b3b1c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa0ac0b53b0 con 0x7fa0ac0d2360 2024-04-15T15:21:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b3b1c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa0ac0b4a80 con 0x7fa0ac15eae0 2024-04-15T15:21:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b10b7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0ac0d2360 0x7fa0ac0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b10b7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0ac0d2360 0x7fa0ac0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35960/0 (socket says 172.21.15.12:35960) 2024-04-15T15:21:04.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b10b7640 1 -- 172.21.15.12:0/3287297969 learned_addr learned my addr 172.21.15.12:0/3287297969 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:04.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.826+0000 7fa0b20b9640 1 --2- 172.21.15.12:0/3287297969 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 0x7fa0ac0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.827+0000 7fa0b10b7640 1 -- 172.21.15.12:0/3287297969 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 msgr2=0x7fa0ac0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:04.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.827+0000 7fa0b10b7640 1 --2- 172.21.15.12:0/3287297969 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 0x7fa0ac0d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.827+0000 7fa0b10b7640 1 -- 172.21.15.12:0/3287297969 >> v1:172.21.15.55:6789/0 conn(0x7fa0ac15e700 legacy=0x7fa0ac0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.827+0000 7fa0b10b7640 1 -- 172.21.15.12:0/3287297969 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ac0b4d60 con 0x7fa0ac0d2360 2024-04-15T15:21:04.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.827+0000 7fa0b10b7640 1 --2- 172.21.15.12:0/3287297969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac0d2360 0x7fa0ac0d2b60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009870 tx=0x7fa0a002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afe53fbf280a55f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b08b6640 1 -- 172.21.15.12:0/3287297969 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0034070 con 0x7fa0ac0d2360 2024-04-15T15:21:04.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b08b6640 1 -- 172.21.15.12:0/3287297969 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a0030840 con 0x7fa0ac0d2360 2024-04-15T15:21:04.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b08b6640 1 -- 172.21.15.12:0/3287297969 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0030c10 con 0x7fa0ac0d2360 2024-04-15T15:21:04.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/3287297969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac0d2360 msgr2=0x7fa0ac0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/3287297969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac0d2360 0x7fa0ac0d2b60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009870 tx=0x7fa0a002b980 comp rx=0 tx=0).stop 2024-04-15T15:21:04.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/3287297969 shutdown_connections 2024-04-15T15:21:04.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/3287297969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac0d2360 0x7fa0ac0d2b60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/3287297969 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 0x7fa0ac0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/3287297969 >> 172.21.15.12:0/3287297969 conn(0x7fa0ac08bf60 msgr2=0x7fa0ac0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/3287297969 shutdown_connections 2024-04-15T15:21:04.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.828+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/3287297969 wait complete. 2024-04-15T15:21:04.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.829+0000 7fa0b3b1c640 1 Processor -- start 2024-04-15T15:21:04.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.829+0000 7fa0b3b1c640 1 -- start start 2024-04-15T15:21:04.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b3b1c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0ac0d2360 0x7fa0ac0b9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b3b1c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac15e700 0x7fa0ac161710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b3b1c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 0x7fa0ac1f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b3b1c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ac0d1d40 con 0x7fa0ac15e700 2024-04-15T15:21:04.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b3b1c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ac0b4450 con 0x7fa0ac0d2360 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b3b1c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa0ac0d1ec0 con 0x7fa0ac15eae0 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b10b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac15e700 0x7fa0ac161710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b10b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac15e700 0x7fa0ac161710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35972/0 (socket says 172.21.15.12:35972) 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b10b7640 1 -- 172.21.15.12:0/1992087868 learned_addr learned my addr 172.21.15.12:0/1992087868 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b18b8640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0ac0d2360 0x7fa0ac0b9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.830+0000 7fa0b20b9640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 0x7fa0ac1f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b10b7640 1 -- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 msgr2=0x7fa0ac1f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b10b7640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 0x7fa0ac1f7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b10b7640 1 -- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0ac0d2360 msgr2=0x7fa0ac0b9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b10b7640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0ac0d2360 0x7fa0ac0b9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b10b7640 1 -- 172.21.15.12:0/1992087868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ac15ef90 con 0x7fa0ac15e700 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b18b8640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0ac0d2360 0x7fa0ac0b9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:04.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b10b7640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac15e700 0x7fa0ac161710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0000c00 tx=0x7fa0a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0aa7fc640 1 -- 172.21.15.12:0/1992087868 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0048070 con 0x7fa0ac15e700 2024-04-15T15:21:04.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0aa7fc640 1 -- 172.21.15.12:0/1992087868 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a0030640 con 0x7fa0ac15e700 2024-04-15T15:21:04.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0ac15f270 con 0x7fa0ac15e700 2024-04-15T15:21:04.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0aa7fc640 1 -- 172.21.15.12:0/1992087868 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0034040 con 0x7fa0ac15e700 2024-04-15T15:21:04.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.831+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0ac0cbd50 con 0x7fa0ac15e700 2024-04-15T15:21:04.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.832+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0ac0689e0 con 0x7fa0ac15e700 2024-04-15T15:21:04.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.832+0000 7fa0aa7fc640 1 -- 172.21.15.12:0/1992087868 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa0a004b070 con 0x7fa0ac15e700 2024-04-15T15:21:04.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.833+0000 7fa0aa7fc640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa09806c920 0x7fa09806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:04.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.833+0000 7fa0aa7fc640 1 -- 172.21.15.12:0/1992087868 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa0a00ad610 con 0x7fa0ac15e700 2024-04-15T15:21:04.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.833+0000 7fa0b18b8640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa09806c920 0x7fa09806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:04.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.834+0000 7fa0b18b8640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa09806c920 0x7fa09806ede0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa09c001330 tx=0x7fa09c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:04.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:04.837+0000 7fa0aa7fc640 1 -- 172.21.15.12:0/1992087868 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0a007b740 con 0x7fa0ac15e700 2024-04-15T15:21:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.067+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0ac0ba500 con 0x7fa0ac15e700 2024-04-15T15:21:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.067+0000 7fa0aa7fc640 1 -- 172.21.15.12:0/1992087868 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7fa0a007fa40 con 0x7fa0ac15e700 2024-04-15T15:21:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:05.070 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-15T15:21:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 2 2024-04-15T15:21:05.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa09806c920 msgr2=0x7fa09806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:05.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa09806c920 0x7fa09806ede0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa09c001330 tx=0x7fa09c009040 comp rx=0 tx=0).stop 2024-04-15T15:21:05.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac15e700 msgr2=0x7fa0ac161710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:05.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac15e700 0x7fa0ac161710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0000c00 tx=0x7fa0a0004730 comp rx=0 tx=0).stop 2024-04-15T15:21:05.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 shutdown_connections 2024-04-15T15:21:05.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0ac15eae0 0x7fa0ac1f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:05.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0ac15e700 0x7fa0ac161710 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:05.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa09806c920 0x7fa09806ede0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:05.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 --2- 172.21.15.12:0/1992087868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0ac0d2360 0x7fa0ac0b9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:05.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 >> 172.21.15.12:0/1992087868 conn(0x7fa0ac08bf60 msgr2=0x7fa0ac0d1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:05.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 shutdown_connections 2024-04-15T15:21:05.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:05.069+0000 7fa0b3b1c640 1 -- 172.21.15.12:0/1992087868 wait complete. 2024-04-15T15:21:07.084 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:21:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.199+0000 7f94093d4640 1 Processor -- start 2024-04-15T15:21:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.199+0000 7f94093d4640 1 -- start start 2024-04-15T15:21:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.199+0000 7f94093d4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f940415e700 0x7f94040b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.199+0000 7f94093d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.199+0000 7f94093d4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94040b47a0 con 0x7f94040d2360 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.199+0000 7f94093d4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f94040b53b0 con 0x7f940415e700 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.199+0000 7f94093d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f94040b4a80 con 0x7f94040d3270 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f94037fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f9402ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f940415e700 0x7f94040b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f94037fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:35542/0 (socket says 172.21.15.12:35542) 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f94027fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f94040d2360 0x7f94040d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:45300/0 (socket says 172.21.15.12:45300) 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f94037fe640 1 -- 172.21.15.12:0/3836635826 learned_addr learned my addr 172.21.15.12:0/3836635826 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f94037fe640 1 -- 172.21.15.12:0/3836635826 >> v2:172.21.15.55:3300/0 conn(0x7f940415e700 msgr2=0x7f94040b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f94037fe640 1 --2- 172.21.15.12:0/3836635826 >> v2:172.21.15.55:3300/0 conn(0x7f940415e700 0x7f94040b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f94037fe640 1 -- 172.21.15.12:0/3836635826 >> v1:172.21.15.55:6789/0 conn(0x7f94040d2360 legacy=0x7f94040d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.200+0000 7f94037fe640 1 -- 172.21.15.12:0/3836635826 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94040b4d60 con 0x7f94040d3270 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.201+0000 7f94037fe640 1 --2- 172.21.15.12:0/3836635826 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cf370 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009e50 tx=0x7f93f402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2404bf5a574b0642 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.201+0000 7f9401ffb640 1 -- 172.21.15.12:0/3836635826 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f402cb00 con 0x7f94040d3270 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.201+0000 7f9401ffb640 1 -- 172.21.15.12:0/3836635826 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f402cca0 con 0x7f94040d3270 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.201+0000 7f9401ffb640 1 -- 172.21.15.12:0/3836635826 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f4034610 con 0x7f94040d3270 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.202+0000 7f94093d4640 1 -- 172.21.15.12:0/3836635826 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 msgr2=0x7f94040cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.202+0000 7f94093d4640 1 --2- 172.21.15.12:0/3836635826 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cf370 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009e50 tx=0x7f93f402bf60 comp rx=0 tx=0).stop 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.202+0000 7f94093d4640 1 -- 172.21.15.12:0/3836635826 shutdown_connections 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.202+0000 7f94093d4640 1 --2- 172.21.15.12:0/3836635826 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cf370 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.202+0000 7f94093d4640 1 --2- 172.21.15.12:0/3836635826 >> v2:172.21.15.55:3300/0 conn(0x7f940415e700 0x7f94040b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.202+0000 7f94093d4640 1 -- 172.21.15.12:0/3836635826 >> 172.21.15.12:0/3836635826 conn(0x7f940408bf60 msgr2=0x7f94040d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.202+0000 7f94093d4640 1 -- 172.21.15.12:0/3836635826 shutdown_connections 2024-04-15T15:21:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.202+0000 7f94093d4640 1 -- 172.21.15.12:0/3836635826 wait complete. 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.203+0000 7f94093d4640 1 Processor -- start 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.203+0000 7f94093d4640 1 -- start start 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f94093d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94040d2360 0x7f94040c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f94093d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f94093d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f940415e700 0x7f94040cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f94093d4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f94040b5d80 con 0x7f94040d2360 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f94093d4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f94040b4450 con 0x7f940415e700 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f94093d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f94040d1d40 con 0x7f94040d3270 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f9402ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94040d2360 0x7f94040c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f9402ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94040d2360 0x7f94040c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35984/0 (socket says 172.21.15.12:35984) 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f9402ffd640 1 -- 172.21.15.12:0/2540304527 learned_addr learned my addr 172.21.15.12:0/2540304527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f94037fe640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f940415e700 0x7f94040cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.204+0000 7f94027fc640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f9402ffd640 1 -- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 msgr2=0x7f94040cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f9402ffd640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f9402ffd640 1 -- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f940415e700 msgr2=0x7f94040cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f9402ffd640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f940415e700 0x7f94040cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f9402ffd640 1 -- 172.21.15.12:0/2540304527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94040c9930 con 0x7f94040d2360 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f9402ffd640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94040d2360 0x7f94040c8fc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00e8a0 tx=0x7f93fc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f93fb7fe640 1 -- 172.21.15.12:0/2540304527 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc017070 con 0x7f94040d2360 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f93fb7fe640 1 -- 172.21.15.12:0/2540304527 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93fc00c940 con 0x7f94040d2360 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94040c9c10 con 0x7f94040d2360 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f93fb7fe640 1 -- 172.21.15.12:0/2540304527 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00ccf0 con 0x7f94040d2360 2024-04-15T15:21:07.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.205+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94040ca1d0 con 0x7f94040d2360 2024-04-15T15:21:07.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.206+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94040689e0 con 0x7f94040d2360 2024-04-15T15:21:07.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.206+0000 7f93fb7fe640 1 -- 172.21.15.12:0/2540304527 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f93fc01a070 con 0x7f94040d2360 2024-04-15T15:21:07.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.207+0000 7f93fb7fe640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f93e406c8f0 0x7f93e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.207+0000 7f93fb7fe640 1 -- 172.21.15.12:0/2540304527 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f93fc08bb30 con 0x7f94040d2360 2024-04-15T15:21:07.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.207+0000 7f94027fc640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f93e406c8f0 0x7f93e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.208+0000 7f94027fc640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f93e406c8f0 0x7f93e406edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94040b7630 tx=0x7f93ec007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.212+0000 7f93fb7fe640 1 -- 172.21.15.12:0/2540304527 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f93fc059c60 con 0x7f94040d2360 2024-04-15T15:21:07.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.437+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94040ba500 con 0x7f94040d2360 2024-04-15T15:21:07.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.438+0000 7f93fb7fe640 1 -- 172.21.15.12:0/2540304527 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3861 (secure 0 0 0) 0x7f93fc05df60 con 0x7f94040d2360 2024-04-15T15:21:07.441 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:07.441 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-15T15:21:07.441 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-04-15T15:21:07.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f93e406c8f0 msgr2=0x7f93e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f93e406c8f0 0x7f93e406edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94040b7630 tx=0x7f93ec007e80 comp rx=0 tx=0).stop 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94040d2360 msgr2=0x7f94040c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94040d2360 0x7f94040c8fc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00e8a0 tx=0x7f93fc00ed60 comp rx=0 tx=0).stop 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 shutdown_connections 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f93e406c8f0 0x7f93e406edb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f940415e700 0x7f94040cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f94040d3270 0x7f94040cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 --2- 172.21.15.12:0/2540304527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94040d2360 0x7f94040c8fc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 >> 172.21.15.12:0/2540304527 conn(0x7f940408bf60 msgr2=0x7f94040d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 shutdown_connections 2024-04-15T15:21:07.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.440+0000 7f94093d4640 1 -- 172.21.15.12:0/2540304527 wait complete. 2024-04-15T15:21:07.453 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:07.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.555+0000 7f7622aba640 1 Processor -- start 2024-04-15T15:21:07.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.556+0000 7f7622aba640 1 -- start start 2024-04-15T15:21:07.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.556+0000 7f7622aba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f761c15ed00 0x7f761c070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.556+0000 7f7622aba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c0cf600 0x7f761c0cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.556+0000 7f7622aba640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f761c06ef70 0x7f761c06f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.556+0000 7f7622aba640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f761c0cffa0 con 0x7f761c06ef70 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.556+0000 7f7622aba640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f761c0ae440 con 0x7f761c15ed00 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.556+0000 7f7622aba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f761c0ae720 con 0x7f761c0cf600 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.556+0000 7f7621057640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f761c06ef70 0x7f761c06f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7621057640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f761c06ef70 0x7f761c06f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35992/0 (socket says 172.21.15.12:35992) 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7621057640 1 -- 172.21.15.12:0/3092477577 learned_addr learned my addr 172.21.15.12:0/3092477577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7620856640 1 --2- 172.21.15.12:0/3092477577 >> v2:172.21.15.55:3300/0 conn(0x7f761c15ed00 0x7f761c070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f761bfff640 1 --2- 172.21.15.12:0/3092477577 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c0cf600 0x7f761c0cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7621057640 1 -- 172.21.15.12:0/3092477577 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c0cf600 msgr2=0x7f761c0cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7621057640 1 --2- 172.21.15.12:0/3092477577 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c0cf600 0x7f761c0cfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7621057640 1 -- 172.21.15.12:0/3092477577 >> v2:172.21.15.55:3300/0 conn(0x7f761c15ed00 msgr2=0x7f761c070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:07.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7621057640 1 --2- 172.21.15.12:0/3092477577 >> v2:172.21.15.55:3300/0 conn(0x7f761c15ed00 0x7f761c070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7621057640 1 -- 172.21.15.12:0/3092477577 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c0aea00 con 0x7f761c06ef70 2024-04-15T15:21:07.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.557+0000 7f7621057640 1 --2- 172.21.15.12:0/3092477577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c06ef70 0x7f761c06f400 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f761c0b4110 tx=0x7f760c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344a2cd79263f5ba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f761b7fe640 1 -- 172.21.15.12:0/3092477577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c035020 con 0x7f761c06ef70 2024-04-15T15:21:07.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f761b7fe640 1 -- 172.21.15.12:0/3092477577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f760c031870 con 0x7f761c06ef70 2024-04-15T15:21:07.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f761b7fe640 1 -- 172.21.15.12:0/3092477577 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c031c40 con 0x7f761c06ef70 2024-04-15T15:21:07.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 -- 172.21.15.12:0/3092477577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c06ef70 msgr2=0x7f761c06f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 --2- 172.21.15.12:0/3092477577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c06ef70 0x7f761c06f400 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f761c0b4110 tx=0x7f760c02b910 comp rx=0 tx=0).stop 2024-04-15T15:21:07.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 -- 172.21.15.12:0/3092477577 shutdown_connections 2024-04-15T15:21:07.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 --2- 172.21.15.12:0/3092477577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c06ef70 0x7f761c06f400 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 --2- 172.21.15.12:0/3092477577 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c0cf600 0x7f761c0cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 --2- 172.21.15.12:0/3092477577 >> v2:172.21.15.55:3300/0 conn(0x7f761c15ed00 0x7f761c070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 -- 172.21.15.12:0/3092477577 >> 172.21.15.12:0/3092477577 conn(0x7f761c09a9d0 msgr2=0x7f761c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:07.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 -- 172.21.15.12:0/3092477577 shutdown_connections 2024-04-15T15:21:07.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.558+0000 7f7622aba640 1 -- 172.21.15.12:0/3092477577 wait complete. 2024-04-15T15:21:07.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.559+0000 7f7622aba640 1 Processor -- start 2024-04-15T15:21:07.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7622aba640 1 -- start start 2024-04-15T15:21:07.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7622aba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c06ef70 0x7f761c1f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7622aba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f761c0cf600 0x7f761c154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7622aba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c15ed00 0x7f761c154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7622aba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f761c0701a0 con 0x7f761c15ed00 2024-04-15T15:21:07.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7622aba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f761c0aefc0 con 0x7f761c0cf600 2024-04-15T15:21:07.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7622aba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f761c070e60 con 0x7f761c06ef70 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7621057640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c15ed00 0x7f761c154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7621057640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c15ed00 0x7f761c154c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36008/0 (socket says 172.21.15.12:36008) 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.560+0000 7f7621057640 1 -- 172.21.15.12:0/1322679943 learned_addr learned my addr 172.21.15.12:0/1322679943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f761bfff640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f761c0cf600 0x7f761c154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7620856640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c06ef70 0x7f761c1f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7621057640 1 -- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c06ef70 msgr2=0x7f761c1f7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7621057640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c06ef70 0x7f761c1f7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7621057640 1 -- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f761c0cf600 msgr2=0x7f761c154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7621057640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f761c0cf600 0x7f761c154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7621057640 1 -- 172.21.15.12:0/1322679943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c158b00 con 0x7f761c15ed00 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7621057640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c15ed00 0x7f761c154c80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f760c02b950 tx=0x7f760c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f76197fa640 1 -- 172.21.15.12:0/1322679943 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c03acb0 con 0x7f761c15ed00 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f76197fa640 1 -- 172.21.15.12:0/1322679943 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f760c03ae50 con 0x7f761c15ed00 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761c0cc040 con 0x7f761c15ed00 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f76197fa640 1 -- 172.21.15.12:0/1322679943 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c044630 con 0x7f761c15ed00 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.561+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f761c0cc600 con 0x7f761c15ed00 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.562+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f761c0690f0 con 0x7f761c15ed00 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.562+0000 7f76197fa640 1 -- 172.21.15.12:0/1322679943 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f760c038020 con 0x7f761c15ed00 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.563+0000 7f76197fa640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f75fc06c8f0 0x7f75fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.563+0000 7f76197fa640 1 -- 172.21.15.12:0/1322679943 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f760c0ad910 con 0x7f761c15ed00 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.563+0000 7f7620856640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f75fc06c8f0 0x7f75fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.564+0000 7f7620856640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f75fc06c8f0 0x7f75fc06edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7614009e90 tx=0x7f761400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.567+0000 7f76197fa640 1 -- 172.21.15.12:0/1322679943 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f760c03f070 con 0x7f761c15ed00 2024-04-15T15:21:07.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.758+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f761c0c88b0 con 0x7f761c15ed00 2024-04-15T15:21:07.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.759+0000 7f76197fa640 1 -- 172.21.15.12:0/1322679943 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7f760c07c120 con 0x7f761c15ed00 2024-04-15T15:21:07.762 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:07.762 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:01.523736+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:07.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f75fc06c8f0 msgr2=0x7f75fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f75fc06c8f0 0x7f75fc06edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7614009e90 tx=0x7f761400b040 comp rx=0 tx=0).stop 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c15ed00 msgr2=0x7f761c154c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c15ed00 0x7f761c154c80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f760c02b950 tx=0x7f760c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 shutdown_connections 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f761c15ed00 0x7f761c154c80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f75fc06c8f0 0x7f75fc06edb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f761c0cf600 0x7f761c154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 --2- 172.21.15.12:0/1322679943 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f761c06ef70 0x7f761c1f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 >> 172.21.15.12:0/1322679943 conn(0x7f761c09a9d0 msgr2=0x7f761c15c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:07.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 shutdown_connections 2024-04-15T15:21:07.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.762+0000 7f7622aba640 1 -- 172.21.15.12:0/1322679943 wait complete. 2024-04-15T15:21:07.775 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:21:07.775 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:07.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.880+0000 7f37d4982640 1 Processor -- start 2024-04-15T15:21:07.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.880+0000 7f37d4982640 1 -- start start 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37d4982640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f37d015e790 0x7f37d00d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37d4982640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d00cfd20 0x7f37d00d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37d4982640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f37d00d0f00 0x7f37d00d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37d4982640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f37d00d2db0 con 0x7f37d00d0f00 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37d4982640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f37d00d2480 con 0x7f37d015e790 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37d4982640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f37d00d2760 con 0x7f37d00cfd20 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ced9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f37d00d0f00 0x7f37d00d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ced9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f37d00d0f00 0x7f37d00d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36016/0 (socket says 172.21.15.12:36016) 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ced9d640 1 -- 172.21.15.12:0/4250523428 learned_addr learned my addr 172.21.15.12:0/4250523428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ce59c640 1 --2- 172.21.15.12:0/4250523428 >> v2:172.21.15.55:3300/0 conn(0x7f37d015e790 0x7f37d00d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37cdd9b640 1 --2- 172.21.15.12:0/4250523428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d00cfd20 0x7f37d00d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ced9d640 1 -- 172.21.15.12:0/4250523428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d00cfd20 msgr2=0x7f37d00d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:07.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ced9d640 1 --2- 172.21.15.12:0/4250523428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d00cfd20 0x7f37d00d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ced9d640 1 -- 172.21.15.12:0/4250523428 >> v2:172.21.15.55:3300/0 conn(0x7f37d015e790 msgr2=0x7f37d00d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ced9d640 1 --2- 172.21.15.12:0/4250523428 >> v2:172.21.15.55:3300/0 conn(0x7f37d015e790 0x7f37d00d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.881+0000 7f37ced9d640 1 -- 172.21.15.12:0/4250523428 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d00d2a40 con 0x7f37d00d0f00 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.882+0000 7f37ce59c640 1 --2- 172.21.15.12:0/4250523428 >> v2:172.21.15.55:3300/0 conn(0x7f37d015e790 0x7f37d00d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.882+0000 7f37cdd9b640 1 --2- 172.21.15.12:0/4250523428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d00cfd20 0x7f37d00d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.882+0000 7f37ced9d640 1 --2- 172.21.15.12:0/4250523428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00d13b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f37d00d4600 tx=0x7f37c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964c54928159cf8f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.882+0000 7f37cd59a640 1 -- 172.21.15.12:0/4250523428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c002ca70 con 0x7f37d00d0f00 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.882+0000 7f37cd59a640 1 -- 172.21.15.12:0/4250523428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37c002cc10 con 0x7f37d00d0f00 2024-04-15T15:21:07.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.882+0000 7f37cd59a640 1 -- 172.21.15.12:0/4250523428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c00343f0 con 0x7f37d00d0f00 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 -- 172.21.15.12:0/4250523428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 msgr2=0x7f37d00d13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 --2- 172.21.15.12:0/4250523428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00d13b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f37d00d4600 tx=0x7f37c002bed0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 -- 172.21.15.12:0/4250523428 shutdown_connections 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 --2- 172.21.15.12:0/4250523428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00d13b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 --2- 172.21.15.12:0/4250523428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d00cfd20 0x7f37d00d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 --2- 172.21.15.12:0/4250523428 >> v2:172.21.15.55:3300/0 conn(0x7f37d015e790 0x7f37d00d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 -- 172.21.15.12:0/4250523428 >> 172.21.15.12:0/4250523428 conn(0x7f37d008bf60 msgr2=0x7f37d00d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 -- 172.21.15.12:0/4250523428 shutdown_connections 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.883+0000 7f37d4982640 1 -- 172.21.15.12:0/4250523428 wait complete. 2024-04-15T15:21:07.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.884+0000 7f37d4982640 1 Processor -- start 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.884+0000 7f37d4982640 1 -- start start 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37d4982640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37d00cfd20 0x7f37d01f8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37d4982640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37d4982640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d015e790 0x7f37d00cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37d4982640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f37d015ecf0 con 0x7f37d00d0f00 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37d4982640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f37d015eb70 con 0x7f37d00cfd20 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37d4982640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f37d00b74f0 con 0x7f37d015e790 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37cdd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37cdd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36020/0 (socket says 172.21.15.12:36020) 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37cdd9b640 1 -- 172.21.15.12:0/47759746 learned_addr learned my addr 172.21.15.12:0/47759746 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37ce59c640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37d00cfd20 0x7f37d01f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.885+0000 7f37ced9d640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d015e790 0x7f37d00cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37cdd9b640 1 -- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d015e790 msgr2=0x7f37d00cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37cdd9b640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d015e790 0x7f37d00cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37cdd9b640 1 -- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37d00cfd20 msgr2=0x7f37d01f8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37cdd9b640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37d00cfd20 0x7f37d01f8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37cdd9b640 1 -- 172.21.15.12:0/47759746 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d00c81c0 con 0x7f37d00d0f00 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37ce59c640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37d00cfd20 0x7f37d01f8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37cdd9b640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00ccf50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f37b800e810 tx=0x7f37b800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37c6ffd640 1 -- 172.21.15.12:0/47759746 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b800f040 con 0x7f37d00d0f00 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37c6ffd640 1 -- 172.21.15.12:0/47759746 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b8004030 con 0x7f37d00d0f00 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d00c84a0 con 0x7f37d00d0f00 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37c6ffd640 1 -- 172.21.15.12:0/47759746 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8004400 con 0x7f37d00d0f00 2024-04-15T15:21:07.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.886+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d00c8a60 con 0x7f37d00d0f00 2024-04-15T15:21:07.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.887+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d00689e0 con 0x7f37d00d0f00 2024-04-15T15:21:07.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.887+0000 7f37c6ffd640 1 -- 172.21.15.12:0/47759746 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f37b800c6d0 con 0x7f37d00d0f00 2024-04-15T15:21:07.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.888+0000 7f37c6ffd640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f37b006c8f0 0x7f37b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:07.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.888+0000 7f37c6ffd640 1 -- 172.21.15.12:0/47759746 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f37b808b500 con 0x7f37d00d0f00 2024-04-15T15:21:07.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.888+0000 7f37ce59c640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f37b006c8f0 0x7f37b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:07.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.889+0000 7f37ce59c640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f37b006c8f0 0x7f37b006edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f37c8006fd0 tx=0x7f37c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:07.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:07.892+0000 7f37c6ffd640 1 -- 172.21.15.12:0/47759746 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f37b8017020 con 0x7f37d00d0f00 2024-04-15T15:21:08.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.075+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f37d00d1ce0 con 0x7f37d00d0f00 2024-04-15T15:21:08.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.075+0000 7f37c6ffd640 1 -- 172.21.15.12:0/47759746 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f37b8059c80 con 0x7f37d00d0f00 2024-04-15T15:21:08.078 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.077+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f37b006c8f0 msgr2=0x7f37b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.077+0000 7f37d4982640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f37b006c8f0 0x7f37b006edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f37c8006fd0 tx=0x7f37c8008040 comp rx=0 tx=0).stop 2024-04-15T15:21:08.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.077+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 msgr2=0x7f37d00ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.077+0000 7f37d4982640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00ccf50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f37b800e810 tx=0x7f37b800ecd0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.078+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 shutdown_connections 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.078+0000 7f37d4982640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f37d015e790 0x7f37d00cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.078+0000 7f37d4982640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37d00d0f00 0x7f37d00ccf50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.078+0000 7f37d4982640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f37b006c8f0 0x7f37b006edb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.078+0000 7f37d4982640 1 --2- 172.21.15.12:0/47759746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37d00cfd20 0x7f37d01f8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.078+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 >> 172.21.15.12:0/47759746 conn(0x7f37d008bf60 msgr2=0x7f37d006e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.078+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 shutdown_connections 2024-04-15T15:21:08.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.078+0000 7f37d4982640 1 -- 172.21.15.12:0/47759746 wait complete. 2024-04-15T15:21:08.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-15T15:21:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.192+0000 7fae1087b640 1 Processor -- start 2024-04-15T15:21:08.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.192+0000 7fae1087b640 1 -- start start 2024-04-15T15:21:08.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.193+0000 7fae1087b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fae0c0701f0 0x7fae0c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.193+0000 7fae1087b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fae0c15fb30 0x7fae0c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.193+0000 7fae1087b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae0c070720 con 0x7fae0c09a9d0 2024-04-15T15:21:08.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.193+0000 7fae1087b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fae0c09ba90 con 0x7fae0c0701f0 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.193+0000 7fae1087b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fae0c0d08c0 con 0x7fae0c15fb30 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.193+0000 7fae09d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fae0c0701f0 0x7fae0c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.194+0000 7fae09d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fae0c0701f0 0x7fae0c160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36032/0 (socket says 172.21.15.12:36032) 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.194+0000 7fae09d9b640 1 -- 172.21.15.12:0/579940240 learned_addr learned my addr 172.21.15.12:0/579940240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.194+0000 7fae0ad9d640 1 --2- 172.21.15.12:0/579940240 >> v2:172.21.15.55:3300/0 conn(0x7fae0c15fb30 0x7fae0c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.194+0000 7fae09d9b640 1 -- 172.21.15.12:0/579940240 >> v2:172.21.15.55:3300/0 conn(0x7fae0c15fb30 msgr2=0x7fae0c0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.194+0000 7fae09d9b640 1 --2- 172.21.15.12:0/579940240 >> v2:172.21.15.55:3300/0 conn(0x7fae0c15fb30 0x7fae0c0d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.194+0000 7fae09d9b640 1 -- 172.21.15.12:0/579940240 >> v1:172.21.15.55:6789/0 conn(0x7fae0c09a9d0 legacy=0x7fae0c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.194+0000 7fae09d9b640 1 -- 172.21.15.12:0/579940240 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0c070a90 con 0x7fae0c0701f0 2024-04-15T15:21:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae09d9b640 1 --2- 172.21.15.12:0/579940240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0701f0 0x7fae0c160420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae0c073740 tx=0x7fadf402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317a9b848b0677da server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae0959a640 1 -- 172.21.15.12:0/579940240 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4034070 con 0x7fae0c0701f0 2024-04-15T15:21:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae0959a640 1 -- 172.21.15.12:0/579940240 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf4030860 con 0x7fae0c0701f0 2024-04-15T15:21:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae0959a640 1 -- 172.21.15.12:0/579940240 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4030c30 con 0x7fae0c0701f0 2024-04-15T15:21:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae1087b640 1 -- 172.21.15.12:0/579940240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0701f0 msgr2=0x7fae0c160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae1087b640 1 --2- 172.21.15.12:0/579940240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0701f0 0x7fae0c160420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae0c073740 tx=0x7fadf402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae1087b640 1 -- 172.21.15.12:0/579940240 shutdown_connections 2024-04-15T15:21:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae1087b640 1 --2- 172.21.15.12:0/579940240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c0701f0 0x7fae0c160420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae1087b640 1 --2- 172.21.15.12:0/579940240 >> v2:172.21.15.55:3300/0 conn(0x7fae0c15fb30 0x7fae0c0d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae1087b640 1 -- 172.21.15.12:0/579940240 >> 172.21.15.12:0/579940240 conn(0x7fae0c0d2460 msgr2=0x7fae0c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae1087b640 1 -- 172.21.15.12:0/579940240 shutdown_connections 2024-04-15T15:21:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.195+0000 7fae1087b640 1 -- 172.21.15.12:0/579940240 wait complete. 2024-04-15T15:21:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.197+0000 7fae1087b640 1 Processor -- start 2024-04-15T15:21:08.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.197+0000 7fae1087b640 1 -- start start 2024-04-15T15:21:08.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.197+0000 7fae1087b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c0701f0 0x7fae0c14db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.197+0000 7fae1087b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c09a9d0 0x7fae0c0c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae1087b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae0c15fb30 0x7fae0c0c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae1087b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c15ffc0 con 0x7fae0c09a9d0 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae1087b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c071050 con 0x7fae0c0701f0 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae1087b640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fae0c0b1c60 con 0x7fae0c15fb30 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c09a9d0 0x7fae0c0c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c09a9d0 0x7fae0c0c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36040/0 (socket says 172.21.15.12:36040) 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 -- 172.21.15.12:0/2407411397 learned_addr learned my addr 172.21.15.12:0/2407411397 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae0a59c640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c0701f0 0x7fae0c14db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae0ad9d640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae0c15fb30 0x7fae0c0c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 -- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae0c15fb30 msgr2=0x7fae0c0c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae0c15fb30 0x7fae0c0c86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 -- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c0701f0 msgr2=0x7fae0c14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c0701f0 0x7fae0c14db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 -- 172.21.15.12:0/2407411397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0c14e080 con 0x7fae0c09a9d0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae09d9b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c09a9d0 0x7fae0c0c81b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fadf4000c00 tx=0x7fadf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae02ffd640 1 -- 172.21.15.12:0/2407411397 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4034040 con 0x7fae0c09a9d0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae02ffd640 1 -- 172.21.15.12:0/2407411397 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf4039d90 con 0x7fae0c09a9d0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae0c14b0a0 con 0x7fae0c09a9d0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae02ffd640 1 -- 172.21.15.12:0/2407411397 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4043610 con 0x7fae0c09a9d0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.198+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae0c14b660 con 0x7fae0c09a9d0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.199+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae0c0689e0 con 0x7fae0c09a9d0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.200+0000 7fae02ffd640 1 -- 172.21.15.12:0/2407411397 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fadf4037070 con 0x7fae0c09a9d0 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.200+0000 7fae02ffd640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadec06c8f0 0x7fadec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.200+0000 7fae02ffd640 1 -- 172.21.15.12:0/2407411397 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fadf40ad890 con 0x7fae0c09a9d0 2024-04-15T15:21:08.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.200+0000 7fae0a59c640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadec06c8f0 0x7fadec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.201+0000 7fae0a59c640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadec06c8f0 0x7fadec06edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fae04001780 tx=0x7fae04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:08.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.205+0000 7fae02ffd640 1 -- 172.21.15.12:0/2407411397 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae0c14e080 con 0x7fae0c09a9d0 2024-04-15T15:21:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.390+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fae0c0b3a40 con 0x7fae0c09a9d0 2024-04-15T15:21:08.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.390+0000 7fae02ffd640 1 -- 172.21.15.12:0/2407411397 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fadf403e070 con 0x7fae0c09a9d0 2024-04-15T15:21:08.393 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:08.393 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-04-15T15:21:08.393 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:21:08.393 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-04-15T15:21:08.393 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-04-15T15:21:08.393 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:21:08.393 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadec06c8f0 msgr2=0x7fadec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadec06c8f0 0x7fadec06edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fae04001780 tx=0x7fae04009040 comp rx=0 tx=0).stop 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c09a9d0 msgr2=0x7fae0c0c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c09a9d0 0x7fae0c0c81b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fadf4000c00 tx=0x7fadf40046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 shutdown_connections 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae0c15fb30 0x7fae0c0c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae0c09a9d0 0x7fae0c0c81b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadec06c8f0 0x7fadec06edb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 --2- 172.21.15.12:0/2407411397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c0701f0 0x7fae0c14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 >> 172.21.15.12:0/2407411397 conn(0x7fae0c0d2460 msgr2=0x7fae0c06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 shutdown_connections 2024-04-15T15:21:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.392+0000 7fae1087b640 1 -- 172.21.15.12:0/2407411397 wait complete. 2024-04-15T15:21:08.405 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-15T15:21:08.406 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-15T15:21:08.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.511+0000 7f3b9eb74640 1 Processor -- start 2024-04-15T15:21:08.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.511+0000 7f3b9eb74640 1 -- start start 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9eb74640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3b9815eb70 0x7f3b980d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9eb74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980cfe80 0x7f3b9806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9eb74640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b980d1070 0x7f3b980d1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9eb74640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3b980d1a40 con 0x7f3b980d1070 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9eb74640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3b980d2c00 con 0x7f3b9815eb70 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9eb74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3b980d2ee0 con 0x7f3b980cfe80 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9d111640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b980d1070 0x7f3b980d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9d111640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b980d1070 0x7f3b980d1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36046/0 (socket says 172.21.15.12:36046) 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9d111640 1 -- 172.21.15.12:0/527430613 learned_addr learned my addr 172.21.15.12:0/527430613 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9c910640 1 --2- 172.21.15.12:0/527430613 >> v2:172.21.15.55:3300/0 conn(0x7f3b9815eb70 0x7f3b980d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b97fff640 1 --2- 172.21.15.12:0/527430613 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980cfe80 0x7f3b9806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9d111640 1 -- 172.21.15.12:0/527430613 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980cfe80 msgr2=0x7f3b9806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:08.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9d111640 1 --2- 172.21.15.12:0/527430613 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980cfe80 0x7f3b9806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9d111640 1 -- 172.21.15.12:0/527430613 >> v2:172.21.15.55:3300/0 conn(0x7f3b9815eb70 msgr2=0x7f3b980d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:08.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9d111640 1 --2- 172.21.15.12:0/527430613 >> v2:172.21.15.55:3300/0 conn(0x7f3b9815eb70 0x7f3b980d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.512+0000 7f3b9d111640 1 -- 172.21.15.12:0/527430613 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b980d31c0 con 0x7f3b980d1070 2024-04-15T15:21:08.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.513+0000 7f3b9d111640 1 --2- 172.21.15.12:0/527430613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b980d1070 0x7f3b980d1500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b9808cf00 tx=0x7f3b9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107167bdc8de8971 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:08.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.513+0000 7f3b977fe640 1 -- 172.21.15.12:0/527430613 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90035020 con 0x7f3b980d1070 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.513+0000 7f3b977fe640 1 -- 172.21.15.12:0/527430613 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b90031870 con 0x7f3b980d1070 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.513+0000 7f3b977fe640 1 -- 172.21.15.12:0/527430613 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90031c40 con 0x7f3b980d1070 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 -- 172.21.15.12:0/527430613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b980d1070 msgr2=0x7f3b980d1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/527430613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b980d1070 0x7f3b980d1500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b9808cf00 tx=0x7f3b9002b910 comp rx=0 tx=0).stop 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 -- 172.21.15.12:0/527430613 shutdown_connections 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/527430613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b980d1070 0x7f3b980d1500 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/527430613 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980cfe80 0x7f3b9806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/527430613 >> v2:172.21.15.55:3300/0 conn(0x7f3b9815eb70 0x7f3b980d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 -- 172.21.15.12:0/527430613 >> 172.21.15.12:0/527430613 conn(0x7f3b980b61d0 msgr2=0x7f3b980b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 -- 172.21.15.12:0/527430613 shutdown_connections 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.514+0000 7f3b9eb74640 1 -- 172.21.15.12:0/527430613 wait complete. 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.515+0000 7f3b9eb74640 1 Processor -- start 2024-04-15T15:21:08.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.515+0000 7f3b9eb74640 1 -- start start 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9eb74640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b980cfe80 0x7f3b980cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9eb74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980d1070 0x7f3b980cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9eb74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b9815eb70 0x7f3b980cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9eb74640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9806d5f0 con 0x7f3b9815eb70 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9eb74640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9815ef50 con 0x7f3b980cfe80 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9eb74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3b9806d770 con 0x7f3b980d1070 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b9815eb70 0x7f3b980cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b9815eb70 0x7f3b980cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36054/0 (socket says 172.21.15.12:36054) 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 -- 172.21.15.12:0/3326629280 learned_addr learned my addr 172.21.15.12:0/3326629280 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9c910640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b980cfe80 0x7f3b980cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b97fff640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980d1070 0x7f3b980cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 -- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980d1070 msgr2=0x7f3b980cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980d1070 0x7f3b980cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 -- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b980cfe80 msgr2=0x7f3b980cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b980cfe80 0x7f3b980cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 -- 172.21.15.12:0/3326629280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b980cf5e0 con 0x7f3b9815eb70 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.516+0000 7f3b9d111640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b9815eb70 0x7f3b980cb790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009960 tx=0x7f3b900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.517+0000 7f3b957fa640 1 -- 172.21.15.12:0/3326629280 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90040070 con 0x7f3b9815eb70 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.517+0000 7f3b957fa640 1 -- 172.21.15.12:0/3326629280 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b90030c20 con 0x7f3b9815eb70 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.517+0000 7f3b957fa640 1 -- 172.21.15.12:0/3326629280 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90035040 con 0x7f3b9815eb70 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.517+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b980cb0d0 con 0x7f3b9815eb70 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.517+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b980ce820 con 0x7f3b9815eb70 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.518+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b980689e0 con 0x7f3b9815eb70 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.518+0000 7f3b957fa640 1 -- 172.21.15.12:0/3326629280 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b90038020 con 0x7f3b9815eb70 2024-04-15T15:21:08.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.518+0000 7f3b957fa640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b7806c8f0 0x7f3b7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:08.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.519+0000 7f3b957fa640 1 -- 172.21.15.12:0/3326629280 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f3b900ad480 con 0x7f3b9815eb70 2024-04-15T15:21:08.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.519+0000 7f3b9c910640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b7806c8f0 0x7f3b7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:08.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.519+0000 7f3b9c910640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b7806c8f0 0x7f3b7806edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3b88011340 tx=0x7f3b8800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:08.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.523+0000 7f3b957fa640 1 -- 172.21.15.12:0/3326629280 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b9007b510 con 0x7f3b9815eb70 2024-04-15T15:21:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:08.712+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3b980d2610 con 0x7f3b9815eb70 2024-04-15T15:21:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.610+0000 7f3b957fa640 1 -- 172.21.15.12:0/3326629280 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f3b9007f810 con 0x7f3b9815eb70 2024-04-15T15:21:09.613 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-04-15T15:21:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.613+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b7806c8f0 msgr2=0x7f3b7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.613+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b7806c8f0 0x7f3b7806edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3b88011340 tx=0x7f3b8800b040 comp rx=0 tx=0).stop 2024-04-15T15:21:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.613+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b9815eb70 msgr2=0x7f3b980cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:09.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b9815eb70 0x7f3b980cb790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009960 tx=0x7f3b900046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 shutdown_connections 2024-04-15T15:21:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b9815eb70 0x7f3b980cb790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b980d1070 0x7f3b980cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b7806c8f0 0x7f3b7806edb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 --2- 172.21.15.12:0/3326629280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b980cfe80 0x7f3b980cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 >> 172.21.15.12:0/3326629280 conn(0x7f3b980b61d0 msgr2=0x7f3b980b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 shutdown_connections 2024-04-15T15:21:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.614+0000 7f3b9eb74640 1 -- 172.21.15.12:0/3326629280 wait complete. 2024-04-15T15:21:09.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-15T15:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.749+0000 7fa99393d640 1 Processor -- start 2024-04-15T15:21:09.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.749+0000 7fa99393d640 1 -- start start 2024-04-15T15:21:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa99393d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa98c09b880 0x7fa98c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa99393d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98c0d05e0 con 0x7fa98c0d0da0 2024-04-15T15:21:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa99393d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98c071260 con 0x7fa98c070940 2024-04-15T15:21:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa99393d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa98c09bce0 con 0x7fa98c09b880 2024-04-15T15:21:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa991eda640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa98c0d0da0 0x7fa98c0d11d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33252/0 (socket says 172.21.15.12:33252) 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa991eda640 1 -- 172.21.15.12:0/3882665893 learned_addr learned my addr 172.21.15.12:0/3882665893 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa990ed8640 1 --2- 172.21.15.12:0/3882665893 >> v2:172.21.15.55:3300/0 conn(0x7fa98c09b880 0x7fa98c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1726607040 0 0) 0x7fa98c0d05e0 con 0x7fa98c0d0da0 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.750+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa978003520 con 0x7fa98c0d0da0 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3208388829 0 0) 0x7fa978003520 con 0x7fa98c0d0da0 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 >> v2:172.21.15.55:3300/0 conn(0x7fa98c09b880 msgr2=0x7fa98c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 --2- 172.21.15.12:0/3882665893 >> v2:172.21.15.55:3300/0 conn(0x7fa98c09b880 0x7fa98c070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 >> v1:172.21.15.55:6789/0 conn(0x7fa98c070940 legacy=0x7fa98c160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c15f720 con 0x7fa98c0d0da0 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fa9840019f0 con 0x7fa98c0d0da0 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 >> v1:172.21.15.12:6789/0 conn(0x7fa98c0d0da0 legacy=0x7fa98c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9780039b0 0x7fa978003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa978004800 0x7fa978004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9780051c0 0x7fa97800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c09bce0 con 0x7fa9780039b0 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa97800c060 con 0x7fa9780051c0 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa97800c260 con 0x7fa978004800 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa9916d9640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9780039b0 0x7fa978003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa991eda640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9780051c0 0x7fa97800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa990ed8640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa978004800 0x7fa978004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa9916d9640 1 -- 172.21.15.12:0/3882665893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa978004800 msgr2=0x7fa978004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa9916d9640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa978004800 0x7fa978004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa9916d9640 1 -- 172.21.15.12:0/3882665893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9780051c0 msgr2=0x7fa97800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa9916d9640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9780051c0 0x7fa97800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa9916d9640 1 -- 172.21.15.12:0/3882665893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa978003520 con 0x7fa9780039b0 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.751+0000 7fa991eda640 1 -- 172.21.15.12:0/3882665893 reap_dead start 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.752+0000 7fa9916d9640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9780039b0 0x7fa978003dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa988001750 tx=0x7fa978005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136b925328f91a90 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:09.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.752+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988001370 con 0x7fa9780039b0 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.752+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9880052d0 con 0x7fa9780039b0 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.752+0000 7fa983fff640 1 -- 172.21.15.12:0/3882665893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988003830 con 0x7fa9780039b0 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.752+0000 7fa99393d640 1 -- 172.21.15.12:0/3882665893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9780039b0 msgr2=0x7fa978003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.752+0000 7fa99393d640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9780039b0 0x7fa978003dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa988001750 tx=0x7fa978005830 comp rx=0 tx=0).stop 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.752+0000 7fa99393d640 1 -- 172.21.15.12:0/3882665893 shutdown_connections 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.753+0000 7fa99393d640 1 --2- 172.21.15.12:0/3882665893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9780039b0 0x7fa978003dd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.753+0000 7fa99393d640 1 -- 172.21.15.12:0/3882665893 >> 172.21.15.12:0/3882665893 conn(0x7fa98c0b3f40 msgr2=0x7fa98c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.753+0000 7fa99393d640 1 -- 172.21.15.12:0/3882665893 shutdown_connections 2024-04-15T15:21:09.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.753+0000 7fa99393d640 1 -- 172.21.15.12:0/3882665893 wait complete. 2024-04-15T15:21:09.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.754+0000 7fa99393d640 1 Processor -- start 2024-04-15T15:21:09.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.754+0000 7fa99393d640 1 -- start start 2024-04-15T15:21:09.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.754+0000 7fa99393d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa9780039b0 0x7fa98c0b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:09.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.754+0000 7fa99393d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c0c7540 0x7fa98c0c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:09.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa99393d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0c65d0 0x7fa98c0c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:09.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa99393d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c0d0880 con 0x7fa98c0c65d0 2024-04-15T15:21:09.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa99393d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c15fce0 con 0x7fa98c0c7540 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa99393d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa98c070d20 con 0x7fa9780039b0 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0c65d0 0x7fa98c0c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0c65d0 0x7fa98c0c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36070/0 (socket says 172.21.15.12:36070) 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 -- 172.21.15.12:0/2171296729 learned_addr learned my addr 172.21.15.12:0/2171296729 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa9916d9640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa9780039b0 0x7fa98c0b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa990ed8640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c0c7540 0x7fa98c0c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 -- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa9780039b0 msgr2=0x7fa98c0b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa9780039b0 0x7fa98c0b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 -- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c0c7540 msgr2=0x7fa98c0c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c0c7540 0x7fa98c0c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 -- 172.21.15.12:0/2171296729 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c15d380 con 0x7fa98c0c65d0 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa990ed8640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c0c7540 0x7fa98c0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:09.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa9916d9640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa9780039b0 0x7fa98c0b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa991eda640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0c65d0 0x7fa98c0c6a60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0737d0 tx=0x7fa984007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa981ffb640 1 -- 172.21.15.12:0/2171296729 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984017020 con 0x7fa98c0c65d0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa981ffb640 1 -- 172.21.15.12:0/2171296729 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa98400aa60 con 0x7fa98c0c65d0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa981ffb640 1 -- 172.21.15.12:0/2171296729 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98400ae50 con 0x7fa98c0c65d0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.755+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa98c15d660 con 0x7fa98c0c65d0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.756+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa98c0b57d0 con 0x7fa98c0c65d0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.757+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa98c0d1d40 con 0x7fa98c0c65d0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.757+0000 7fa981ffb640 1 -- 172.21.15.12:0/2171296729 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa98401a020 con 0x7fa98c0c65d0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.757+0000 7fa981ffb640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa97006ca90 0x7fa97006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.757+0000 7fa981ffb640 1 -- 172.21.15.12:0/2171296729 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fa98408c0a0 con 0x7fa98c0c65d0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.758+0000 7fa9916d9640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa97006ca90 0x7fa97006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:09.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.758+0000 7fa9916d9640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa97006ca90 0x7fa97006ef50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa988033000 tx=0x7fa988009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:09.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.762+0000 7fa981ffb640 1 -- 172.21.15.12:0/2171296729 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa98c0d1d40 con 0x7fa98c0c65d0 2024-04-15T15:21:09.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:09.947+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fa98c0689e0 con 0x7fa98c0c65d0 2024-04-15T15:21:10.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.618+0000 7fa981ffb640 1 -- 172.21.15.12:0/2171296729 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7fa984021020 con 0x7fa98c0c65d0 2024-04-15T15:21:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-04-15T15:21:10.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa97006ca90 msgr2=0x7fa97006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:10.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa97006ca90 0x7fa97006ef50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa988033000 tx=0x7fa988009030 comp rx=0 tx=0).stop 2024-04-15T15:21:10.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0c65d0 msgr2=0x7fa98c0c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0c65d0 0x7fa98c0c6a60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0737d0 tx=0x7fa984007a90 comp rx=0 tx=0).stop 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 shutdown_connections 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa98c0c65d0 0x7fa98c0c6a60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa97006ca90 0x7fa97006ef50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c0c7540 0x7fa98c0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 --2- 172.21.15.12:0/2171296729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa9780039b0 0x7fa98c0b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 >> 172.21.15.12:0/2171296729 conn(0x7fa98c0b3f40 msgr2=0x7fa98c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 shutdown_connections 2024-04-15T15:21:10.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.621+0000 7fa99393d640 1 -- 172.21.15.12:0/2171296729 wait complete. 2024-04-15T15:21:10.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-15T15:21:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.758+0000 7f4ffc8c5640 1 Processor -- start 2024-04-15T15:21:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.758+0000 7f4ffc8c5640 1 -- start start 2024-04-15T15:21:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ffc8c5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4ff80d3480 0x7f4ff80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ffc8c5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ff80b6790 0x7f4ff80b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ffc8c5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff80cff70 con 0x7f4ff815ea40 2024-04-15T15:21:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ffc8c5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff80d0250 con 0x7f4ff80b6790 2024-04-15T15:21:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ffc8c5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff80d0530 con 0x7f4ff80d3480 2024-04-15T15:21:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ff6d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ff80b6790 0x7f4ff80b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ff6d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ff80b6790 0x7f4ff80b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36082/0 (socket says 172.21.15.12:36082) 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ff6d9d640 1 -- 172.21.15.12:0/1163545131 learned_addr learned my addr 172.21.15.12:0/1163545131 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4fedd9b640 1 --2- 172.21.15.12:0/1163545131 >> v2:172.21.15.55:3300/0 conn(0x7f4ff80d3480 0x7f4ff80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ff6d9d640 1 -- 172.21.15.12:0/1163545131 >> v2:172.21.15.55:3300/0 conn(0x7f4ff80d3480 msgr2=0x7f4ff80b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ff6d9d640 1 --2- 172.21.15.12:0/1163545131 >> v2:172.21.15.55:3300/0 conn(0x7f4ff80d3480 0x7f4ff80b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ff6d9d640 1 -- 172.21.15.12:0/1163545131 >> v1:172.21.15.55:6789/0 conn(0x7f4ff815ea40 legacy=0x7f4ff80b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4ff6d9d640 1 -- 172.21.15.12:0/1163545131 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff80d0810 con 0x7f4ff80b6790 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.759+0000 7f4fedd9b640 1 --2- 172.21.15.12:0/1163545131 >> v2:172.21.15.55:3300/0 conn(0x7f4ff80d3480 0x7f4ff80b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.760+0000 7f4ff6d9d640 1 --2- 172.21.15.12:0/1163545131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80b6790 0x7f4ff80b3e80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ff808cc60 tx=0x7f4ff002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a1142017dee95c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.760+0000 7f4ff5d9b640 1 -- 172.21.15.12:0/1163545131 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff002cbb0 con 0x7f4ff80b6790 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.760+0000 7f4ff5d9b640 1 -- 172.21.15.12:0/1163545131 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff002cd50 con 0x7f4ff80b6790 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.760+0000 7f4ff5d9b640 1 -- 172.21.15.12:0/1163545131 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0034610 con 0x7f4ff80b6790 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.761+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/1163545131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80b6790 msgr2=0x7f4ff80b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.761+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/1163545131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80b6790 0x7f4ff80b3e80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ff808cc60 tx=0x7f4ff002c010 comp rx=0 tx=0).stop 2024-04-15T15:21:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.761+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/1163545131 shutdown_connections 2024-04-15T15:21:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.761+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/1163545131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80b6790 0x7f4ff80b3e80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.761+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/1163545131 >> v2:172.21.15.55:3300/0 conn(0x7f4ff80d3480 0x7f4ff80b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.761+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/1163545131 >> 172.21.15.12:0/1163545131 conn(0x7f4ff808bf60 msgr2=0x7f4ff80d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.761+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/1163545131 shutdown_connections 2024-04-15T15:21:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.761+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/1163545131 wait complete. 2024-04-15T15:21:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.762+0000 7f4ffc8c5640 1 Processor -- start 2024-04-15T15:21:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.762+0000 7f4ffc8c5640 1 -- start start 2024-04-15T15:21:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ffc8c5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff80b6790 0x7f4ff80b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ffc8c5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4ff80d3480 0x7f4ff80b9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ffc8c5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff815ea40 0x7f4ff8156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ffc8c5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff80b3a40 con 0x7f4ff815ea40 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ffc8c5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff80b38c0 con 0x7f4ff80b6790 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ffc8c5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff815ee20 con 0x7f4ff80d3480 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff815ea40 0x7f4ff8156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff815ea40 0x7f4ff8156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36094/0 (socket says 172.21.15.12:36094) 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 -- 172.21.15.12:0/2858088529 learned_addr learned my addr 172.21.15.12:0/2858088529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff659c640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff80b6790 0x7f4ff80b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4fedd9b640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4ff80d3480 0x7f4ff80b9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 -- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4ff80d3480 msgr2=0x7f4ff80b9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4ff80d3480 0x7f4ff80b9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 -- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff80b6790 msgr2=0x7f4ff80b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff80b6790 0x7f4ff80b2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 -- 172.21.15.12:0/2858088529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff80b2fd0 con 0x7f4ff815ea40 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff659c640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff80b6790 0x7f4ff80b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.763+0000 7f4ff6d9d640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff815ea40 0x7f4ff8156790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0036040 tx=0x7f4ff0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.764+0000 7f4fef7fe640 1 -- 172.21.15.12:0/2858088529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0043020 con 0x7f4ff815ea40 2024-04-15T15:21:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.764+0000 7f4fef7fe640 1 -- 172.21.15.12:0/2858088529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff003e450 con 0x7f4ff815ea40 2024-04-15T15:21:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.764+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff80ba3b0 con 0x7f4ff815ea40 2024-04-15T15:21:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.764+0000 7f4fef7fe640 1 -- 172.21.15.12:0/2858088529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0009510 con 0x7f4ff815ea40 2024-04-15T15:21:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.764+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff81550b0 con 0x7f4ff815ea40 2024-04-15T15:21:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.765+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff80735d0 con 0x7f4ff815ea40 2024-04-15T15:21:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.765+0000 7f4fef7fe640 1 -- 172.21.15.12:0/2858088529 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ff0046020 con 0x7f4ff815ea40 2024-04-15T15:21:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.765+0000 7f4fef7fe640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4fd806c920 0x7f4fd806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.766+0000 7f4fef7fe640 1 -- 172.21.15.12:0/2858088529 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f4ff00ae2b0 con 0x7f4ff815ea40 2024-04-15T15:21:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.766+0000 7f4ff659c640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4fd806c920 0x7f4fd806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.766+0000 7f4ff659c640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4fd806c920 0x7f4fd806ede0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4fe80015e0 tx=0x7f4fe8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.770+0000 7f4fef7fe640 1 -- 172.21.15.12:0/2858088529 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ff80735d0 con 0x7f4ff815ea40 2024-04-15T15:21:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:10.998+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4ff81e94c0 con 0x7f4ff815ea40 2024-04-15T15:21:11.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.625+0000 7f4fef7fe640 1 -- 172.21.15.12:0/2858088529 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f4ff004d030 con 0x7f4ff815ea40 2024-04-15T15:21:11.627 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 2 and data pool 3 2024-04-15T15:21:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4fd806c920 msgr2=0x7f4fd806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4fd806c920 0x7f4fd806ede0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4fe80015e0 tx=0x7f4fe8009040 comp rx=0 tx=0).stop 2024-04-15T15:21:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff815ea40 msgr2=0x7f4ff8156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff815ea40 0x7f4ff8156790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0036040 tx=0x7f4ff0004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 shutdown_connections 2024-04-15T15:21:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff815ea40 0x7f4ff8156790 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4ff80d3480 0x7f4ff80b9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4fd806c920 0x7f4fd806ede0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 --2- 172.21.15.12:0/2858088529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff80b6790 0x7f4ff80b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.627+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 >> 172.21.15.12:0/2858088529 conn(0x7f4ff808bf60 msgr2=0x7f4ff815dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.628+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 shutdown_connections 2024-04-15T15:21:11.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.628+0000 7f4ffc8c5640 1 -- 172.21.15.12:0/2858088529 wait complete. 2024-04-15T15:21:11.659 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-15T15:21:11.660 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-15T15:21:11.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.773+0000 7f63b5f74640 1 Processor -- start 2024-04-15T15:21:11.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.773+0000 7f63b5f74640 1 -- start start 2024-04-15T15:21:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63b5f74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b00d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63b5f74640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b00d2340 con 0x7f63b00d1590 2024-04-15T15:21:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63b5f74640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b00d2620 con 0x7f63b00cfcb0 2024-04-15T15:21:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63b5f74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f63b015eca0 con 0x7f63b015e8c0 2024-04-15T15:21:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63affff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f63b00d1590 0x7f63b006c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33260/0 (socket says 172.21.15.12:33260) 2024-04-15T15:21:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63affff640 1 -- 172.21.15.12:0/3012233704 learned_addr learned my addr 172.21.15.12:0/3012233704 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63af7fe640 1 --2- 172.21.15.12:0/3012233704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b00d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63ae7fc640 1 -- 172.21.15.12:0/3012233704 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2754620770 0 0) 0x7f63b00d2340 con 0x7f63b00d1590 2024-04-15T15:21:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63ae7fc640 1 -- 172.21.15.12:0/3012233704 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6398003520 con 0x7f63b00d1590 2024-04-15T15:21:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63ae7fc640 1 -- 172.21.15.12:0/3012233704 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 970236564 0 0) 0x7f63b00d2620 con 0x7f63b00cfcb0 2024-04-15T15:21:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63ae7fc640 1 -- 172.21.15.12:0/3012233704 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63b00d2340 con 0x7f63b00cfcb0 2024-04-15T15:21:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63af7fe640 1 -- 172.21.15.12:0/3012233704 >> v1:172.21.15.55:6789/0 conn(0x7f63b00cfcb0 legacy=0x7f63b00d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63af7fe640 1 -- 172.21.15.12:0/3012233704 >> v1:172.21.15.12:6789/0 conn(0x7f63b00d1590 legacy=0x7f63b006c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.774+0000 7f63af7fe640 1 -- 172.21.15.12:0/3012233704 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b00d2900 con 0x7f63b015e8c0 2024-04-15T15:21:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.775+0000 7f63af7fe640 1 --2- 172.21.15.12:0/3012233704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b00d3100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f63a00097d0 tx=0x7f63a002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b528e9444d3a284 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.775+0000 7f63ae7fc640 1 -- 172.21.15.12:0/3012233704 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0035020 con 0x7f63b015e8c0 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.775+0000 7f63ae7fc640 1 -- 172.21.15.12:0/3012233704 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a0031890 con 0x7f63b015e8c0 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.775+0000 7f63ae7fc640 1 -- 172.21.15.12:0/3012233704 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0031c40 con 0x7f63b015e8c0 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.776+0000 7f63b5f74640 1 -- 172.21.15.12:0/3012233704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 msgr2=0x7f63b00d3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.776+0000 7f63b5f74640 1 --2- 172.21.15.12:0/3012233704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b00d3100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f63a00097d0 tx=0x7f63a002b930 comp rx=0 tx=0).stop 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.776+0000 7f63b5f74640 1 -- 172.21.15.12:0/3012233704 shutdown_connections 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.776+0000 7f63b5f74640 1 --2- 172.21.15.12:0/3012233704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b00d3100 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.776+0000 7f63b5f74640 1 -- 172.21.15.12:0/3012233704 >> 172.21.15.12:0/3012233704 conn(0x7f63b008bf60 msgr2=0x7f63b00d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.776+0000 7f63b5f74640 1 -- 172.21.15.12:0/3012233704 shutdown_connections 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.776+0000 7f63b5f74640 1 -- 172.21.15.12:0/3012233704 wait complete. 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.777+0000 7f63b5f74640 1 Processor -- start 2024-04-15T15:21:11.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.777+0000 7f63b5f74640 1 -- start start 2024-04-15T15:21:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63b5f74640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63b00cfcb0 0x7f63b00c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63b5f74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63b00d1590 0x7f63b00b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63b5f74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b0156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63b5f74640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f63b00d4f60 con 0x7f63b00d1590 2024-04-15T15:21:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63b5f74640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f63b00d4de0 con 0x7f63b00cfcb0 2024-04-15T15:21:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63b5f74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f63b00d0110 con 0x7f63b015e8c0 2024-04-15T15:21:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63aeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63b00d1590 0x7f63b00b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:11.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63aeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63b00d1590 0x7f63b00b82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36110/0 (socket says 172.21.15.12:36110) 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63aeffd640 1 -- 172.21.15.12:0/2894750400 learned_addr learned my addr 172.21.15.12:0/2894750400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63af7fe640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63b00cfcb0 0x7f63b00c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63affff640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b0156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63aeffd640 1 -- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 msgr2=0x7f63b0156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63aeffd640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b0156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63aeffd640 1 -- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63b00cfcb0 msgr2=0x7f63b00c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63aeffd640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63b00cfcb0 0x7f63b00c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.778+0000 7f63aeffd640 1 -- 172.21.15.12:0/2894750400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b00b8a10 con 0x7f63b00d1590 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.779+0000 7f63aeffd640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63b00d1590 0x7f63b00b82e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f63a400c490 tx=0x7f63a400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.779+0000 7f6397fff640 1 -- 172.21.15.12:0/2894750400 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a40052d0 con 0x7f63b00d1590 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.779+0000 7f6397fff640 1 -- 172.21.15.12:0/2894750400 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a4005470 con 0x7f63b00d1590 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.779+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b00cbcf0 con 0x7f63b00d1590 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.779+0000 7f6397fff640 1 -- 172.21.15.12:0/2894750400 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4012850 con 0x7f63b00d1590 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.779+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b00cc2b0 con 0x7f63b00d1590 2024-04-15T15:21:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.780+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63b00cbae0 con 0x7f63b00d1590 2024-04-15T15:21:11.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.780+0000 7f6397fff640 1 -- 172.21.15.12:0/2894750400 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f63a40129f0 con 0x7f63b00d1590 2024-04-15T15:21:11.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.781+0000 7f6397fff640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f639c06c8f0 0x7f639c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:11.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.781+0000 7f6397fff640 1 -- 172.21.15.12:0/2894750400 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f63a408b650 con 0x7f63b00d1590 2024-04-15T15:21:11.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.781+0000 7f63af7fe640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f639c06c8f0 0x7f639c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:11.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.781+0000 7f63af7fe640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f639c06c8f0 0x7f639c06edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63a0036000 tx=0x7f63a003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:11.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.785+0000 7f6397fff640 1 -- 172.21.15.12:0/2894750400 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f63a40594c0 con 0x7f63b00d1590 2024-04-15T15:21:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:11.979+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f63b00689e0 con 0x7f63b00d1590 2024-04-15T15:21:12.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.626+0000 7f6397fff640 1 -- 172.21.15.12:0/2894750400 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f63a405d7c0 con 0x7f63b00d1590 2024-04-15T15:21:12.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.629+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f639c06c8f0 msgr2=0x7f639c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:12.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f639c06c8f0 0x7f639c06edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63a0036000 tx=0x7f63a003c040 comp rx=0 tx=0).stop 2024-04-15T15:21:12.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63b00d1590 msgr2=0x7f63b00b82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:12.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63b00d1590 0x7f63b00b82e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f63a400c490 tx=0x7f63a400c950 comp rx=0 tx=0).stop 2024-04-15T15:21:12.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 shutdown_connections 2024-04-15T15:21:12.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63b015e8c0 0x7f63b0156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63b00d1590 0x7f63b00b82e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f639c06c8f0 0x7f639c06edb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 --2- 172.21.15.12:0/2894750400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63b00cfcb0 0x7f63b00c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 >> 172.21.15.12:0/2894750400 conn(0x7f63b008bf60 msgr2=0x7f63b00b3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:12.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 shutdown_connections 2024-04-15T15:21:12.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.630+0000 7f63b5f74640 1 -- 172.21.15.12:0/2894750400 wait complete. 2024-04-15T15:21:12.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-15T15:21:12.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.762+0000 7f99a4d32640 1 Processor -- start 2024-04-15T15:21:12.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.762+0000 7f99a4d32640 1 -- start start 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.762+0000 7f99a4d32640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f99a0162000 0x7f99a00d3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.762+0000 7f99a4d32640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99a00d1080 0x7f99a00d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.762+0000 7f99a4d32640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99a00d51b0 con 0x7f99a00b3610 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f99a4d32640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f99a00d19f0 con 0x7f99a00d1080 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f99a4d32640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f99a00cf3e0 con 0x7f99a0162000 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f999ed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99a00d1080 0x7f99a00d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f9995d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f99a00b3610 0x7f99a00b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58334/0 (socket says 172.21.15.12:58334) 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f999ed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99a00d1080 0x7f99a00d14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40896/0 (socket says 172.21.15.12:40896) 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f9995d9b640 1 -- 172.21.15.12:0/1710217436 learned_addr learned my addr 172.21.15.12:0/1710217436 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f999e59c640 1 --2- 172.21.15.12:0/1710217436 >> v2:172.21.15.55:3300/0 conn(0x7f99a0162000 0x7f99a00d3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f999ed9d640 1 -- 172.21.15.12:0/1710217436 >> v2:172.21.15.55:3300/0 conn(0x7f99a0162000 msgr2=0x7f99a00d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f999ed9d640 1 --2- 172.21.15.12:0/1710217436 >> v2:172.21.15.55:3300/0 conn(0x7f99a0162000 0x7f99a00d3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f999ed9d640 1 -- 172.21.15.12:0/1710217436 >> v1:172.21.15.12:6789/0 conn(0x7f99a00b3610 legacy=0x7f99a00b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f999ed9d640 1 -- 172.21.15.12:0/1710217436 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a00cf6c0 con 0x7f99a00d1080 2024-04-15T15:21:12.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.763+0000 7f999dd9b640 1 -- 172.21.15.12:0/1710217436 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1426105676 0 0) 0x7f99a00d51b0 con 0x7f99a00b3610 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f999ed9d640 1 --2- 172.21.15.12:0/1710217436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a00d1080 0x7f99a00d14b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f99a008cdc0 tx=0x7f999802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abaabf147f2576b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f999dd9b640 1 -- 172.21.15.12:0/1710217436 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998039070 con 0x7f99a00d1080 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f999dd9b640 1 -- 172.21.15.12:0/1710217436 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f999802cc80 con 0x7f99a00d1080 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f999dd9b640 1 -- 172.21.15.12:0/1710217436 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998034660 con 0x7f99a00d1080 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f99a4d32640 1 -- 172.21.15.12:0/1710217436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a00d1080 msgr2=0x7f99a00d14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f99a4d32640 1 --2- 172.21.15.12:0/1710217436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a00d1080 0x7f99a00d14b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f99a008cdc0 tx=0x7f999802bed0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f99a4d32640 1 -- 172.21.15.12:0/1710217436 shutdown_connections 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f99a4d32640 1 --2- 172.21.15.12:0/1710217436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a00d1080 0x7f99a00d14b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f99a4d32640 1 --2- 172.21.15.12:0/1710217436 >> v2:172.21.15.55:3300/0 conn(0x7f99a0162000 0x7f99a00d3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f99a4d32640 1 -- 172.21.15.12:0/1710217436 >> 172.21.15.12:0/1710217436 conn(0x7f99a008bf60 msgr2=0x7f99a00d4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f99a4d32640 1 -- 172.21.15.12:0/1710217436 shutdown_connections 2024-04-15T15:21:12.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.764+0000 7f99a4d32640 1 -- 172.21.15.12:0/1710217436 wait complete. 2024-04-15T15:21:12.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.766+0000 7f99a4d32640 1 Processor -- start 2024-04-15T15:21:12.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.766+0000 7f99a4d32640 1 -- start start 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99a4d32640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99a00b3610 0x7f99a00caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99a4d32640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99a00d1080 0x7f99a00b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99a4d32640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a0162000 0x7f99a009f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99a4d32640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f99a00d5490 con 0x7f99a0162000 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99a4d32640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f99a00b46e0 con 0x7f99a00d1080 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99a4d32640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f99a01623e0 con 0x7f99a00b3610 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a0162000 0x7f99a009f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a0162000 0x7f99a009f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40908/0 (socket says 172.21.15.12:40908) 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 -- 172.21.15.12:0/3058383831 learned_addr learned my addr 172.21.15.12:0/3058383831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f9995d9b640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99a00d1080 0x7f99a00b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999e59c640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99a00b3610 0x7f99a00caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 -- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99a00b3610 msgr2=0x7f99a00caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99a00b3610 0x7f99a00caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 -- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99a00d1080 msgr2=0x7f99a00b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:12.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99a00d1080 0x7f99a00b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:12.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 -- 172.21.15.12:0/3058383831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a00b2ba0 con 0x7f99a0162000 2024-04-15T15:21:12.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f9995d9b640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99a00d1080 0x7f99a00b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:12.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f999ed9d640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a0162000 0x7f99a009f3a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f999802c4e0 tx=0x7f9998004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:12.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99977fe640 1 -- 172.21.15.12:0/3058383831 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998039050 con 0x7f99a0162000 2024-04-15T15:21:12.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99977fe640 1 -- 172.21.15.12:0/3058383831 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9998045070 con 0x7f99a0162000 2024-04-15T15:21:12.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a00cc830 con 0x7f99a0162000 2024-04-15T15:21:12.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.767+0000 7f99977fe640 1 -- 172.21.15.12:0/3058383831 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999803e830 con 0x7f99a0162000 2024-04-15T15:21:12.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.768+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a00ccdf0 con 0x7f99a0162000 2024-04-15T15:21:12.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.769+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a00689e0 con 0x7f99a0162000 2024-04-15T15:21:12.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.769+0000 7f99977fe640 1 -- 172.21.15.12:0/3058383831 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f999803e9d0 con 0x7f99a0162000 2024-04-15T15:21:12.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.769+0000 7f99977fe640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f998006c8f0 0x7f998006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:12.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.770+0000 7f99977fe640 1 -- 172.21.15.12:0/3058383831 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7f99980aea10 con 0x7f99a0162000 2024-04-15T15:21:12.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.770+0000 7f999e59c640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f998006c8f0 0x7f998006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:12.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.770+0000 7f999e59c640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f998006c8f0 0x7f998006edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9990006fd0 tx=0x7f9990008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:12.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.774+0000 7f99977fe640 1 -- 172.21.15.12:0/3058383831 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9998032e20 con 0x7f99a0162000 2024-04-15T15:21:12.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:12.959+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f99a008c6d0 con 0x7f99a0162000 2024-04-15T15:21:14.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.666+0000 7f99977fe640 1 -- 172.21.15.12:0/3058383831 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7f999807ce20 con 0x7f99a0162000 2024-04-15T15:21:14.669 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data_ec' created 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f998006c8f0 msgr2=0x7f998006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f998006c8f0 0x7f998006edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9990006fd0 tx=0x7f9990008040 comp rx=0 tx=0).stop 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a0162000 msgr2=0x7f99a009f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a0162000 0x7f99a009f3a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f999802c4e0 tx=0x7f9998004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 shutdown_connections 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a0162000 0x7f99a009f3a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f998006c8f0 0x7f998006edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99a00d1080 0x7f99a00b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 --2- 172.21.15.12:0/3058383831 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99a00b3610 0x7f99a00caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:14.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 >> 172.21.15.12:0/3058383831 conn(0x7f99a008bf60 msgr2=0x7f99a006e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:14.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.669+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 shutdown_connections 2024-04-15T15:21:14.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.670+0000 7f99a4d32640 1 -- 172.21.15.12:0/3058383831 wait complete. 2024-04-15T15:21:14.691 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-15T15:21:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.803+0000 7f0142b59640 1 Processor -- start 2024-04-15T15:21:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.803+0000 7f0142b59640 1 -- start start 2024-04-15T15:21:14.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f0142b59640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f013c15b6b0 0x7f013c0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:14.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f0142b59640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013c0cde00 con 0x7f013c0d0a60 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f0142b59640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013c0ce0e0 con 0x7f013c0b3b90 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f0142b59640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f013c0b3fe0 con 0x7f013c15b6b0 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f01408f5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f013c15b6b0 0x7f013c0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f013bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f013c0d0a60 0x7f013c0b3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58340/0 (socket says 172.21.15.12:58340) 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f01408f5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f013c15b6b0 0x7f013c0d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40920/0 (socket says 172.21.15.12:40920) 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f013bfff640 1 -- 172.21.15.12:0/245638905 learned_addr learned my addr 172.21.15.12:0/245638905 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f01408f5640 1 -- 172.21.15.12:0/245638905 >> v1:172.21.15.55:6789/0 conn(0x7f013c0b3b90 legacy=0x7f013c0ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f01408f5640 1 -- 172.21.15.12:0/245638905 >> v1:172.21.15.12:6789/0 conn(0x7f013c0d0a60 legacy=0x7f013c0b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.804+0000 7f01408f5640 1 -- 172.21.15.12:0/245638905 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c0ce3c0 con 0x7f013c15b6b0 2024-04-15T15:21:14.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.805+0000 7f01408f5640 1 --2- 172.21.15.12:0/245638905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c15b6b0 0x7f013c0d0520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01300097a0 tx=0x7f013002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25eb78f974f94ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:14.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.805+0000 7f013b7fe640 1 -- 172.21.15.12:0/245638905 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013002f810 con 0x7f013c15b6b0 2024-04-15T15:21:14.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.805+0000 7f013b7fe640 1 -- 172.21.15.12:0/245638905 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013002f9b0 con 0x7f013c15b6b0 2024-04-15T15:21:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.805+0000 7f013b7fe640 1 -- 172.21.15.12:0/245638905 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013002fd80 con 0x7f013c15b6b0 2024-04-15T15:21:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.805+0000 7f0142b59640 1 -- 172.21.15.12:0/245638905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c15b6b0 msgr2=0x7f013c0d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.805+0000 7f0142b59640 1 --2- 172.21.15.12:0/245638905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c15b6b0 0x7f013c0d0520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01300097a0 tx=0x7f013002b9f0 comp rx=0 tx=0).stop 2024-04-15T15:21:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.806+0000 7f0142b59640 1 -- 172.21.15.12:0/245638905 shutdown_connections 2024-04-15T15:21:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.806+0000 7f0142b59640 1 --2- 172.21.15.12:0/245638905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c15b6b0 0x7f013c0d0520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.806+0000 7f0142b59640 1 -- 172.21.15.12:0/245638905 >> 172.21.15.12:0/245638905 conn(0x7f013c06bbc0 msgr2=0x7f013c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.806+0000 7f0142b59640 1 -- 172.21.15.12:0/245638905 shutdown_connections 2024-04-15T15:21:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.806+0000 7f0142b59640 1 -- 172.21.15.12:0/245638905 wait complete. 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.807+0000 7f0142b59640 1 Processor -- start 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.807+0000 7f0142b59640 1 -- start start 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f0142b59640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f013c0b3b90 0x7f013c15e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f0142b59640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f013c0d0a60 0x7f013c1f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f0142b59640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c0afe40 0x7f013c15baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f0142b59640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f013c0cb040 con 0x7f013c0afe40 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f0142b59640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f013c0ce980 con 0x7f013c0d0a60 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f0142b59640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f013c0cb1c0 con 0x7f013c0b3b90 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c0afe40 0x7f013c15baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c0afe40 0x7f013c15baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40924/0 (socket says 172.21.15.12:40924) 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 -- 172.21.15.12:0/761169125 learned_addr learned my addr 172.21.15.12:0/761169125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:14.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f013bfff640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f013c0d0a60 0x7f013c1f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 -- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f013c0b3b90 msgr2=0x7f013c15e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01408f5640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f013c0b3b90 0x7f013c15e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f013c0b3b90 0x7f013c15e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 -- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f013c0d0a60 msgr2=0x7f013c1f7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f013c0d0a60 0x7f013c1f7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 -- 172.21.15.12:0/761169125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c1f8200 con 0x7f013c0afe40 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f013bfff640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f013c0d0a60 0x7f013c1f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01410f6640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c0afe40 0x7f013c15baa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f013400c760 tx=0x7f013400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01397fa640 1 -- 172.21.15.12:0/761169125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134003b70 con 0x7f013c0afe40 2024-04-15T15:21:14.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01397fa640 1 -- 172.21.15.12:0/761169125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0134003d30 con 0x7f013c0afe40 2024-04-15T15:21:14.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013c1f84e0 con 0x7f013c0afe40 2024-04-15T15:21:14.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f01397fa640 1 -- 172.21.15.12:0/761169125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013400aa90 con 0x7f013c0afe40 2024-04-15T15:21:14.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.808+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013c0b1b70 con 0x7f013c0afe40 2024-04-15T15:21:14.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.809+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013c0689e0 con 0x7f013c0afe40 2024-04-15T15:21:14.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.810+0000 7f01397fa640 1 -- 172.21.15.12:0/761169125 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f013400ac30 con 0x7f013c0afe40 2024-04-15T15:21:14.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.810+0000 7f01397fa640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f012006c920 0x7f012006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:14.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.810+0000 7f01397fa640 1 -- 172.21.15.12:0/761169125 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f013408bc20 con 0x7f013c0afe40 2024-04-15T15:21:14.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.811+0000 7f01408f5640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f012006c920 0x7f012006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:14.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.811+0000 7f01408f5640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f012006c920 0x7f012006ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0130036000 tx=0x7f0130039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:14.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.815+0000 7f01397fa640 1 -- 172.21.15.12:0/761169125 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0134059dc0 con 0x7f013c0afe40 2024-04-15T15:21:15.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:14.999+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f013c0cc610 con 0x7f013c0afe40 2024-04-15T15:21:15.168 INFO:tasks.ceph.mds.d.smithi055.stderr:2024-04-15T15:21:15.166+0000 7f06bd4b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:21:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.677+0000 7f01397fa640 1 -- 172.21.15.12:0/761169125 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7f01340596c0 con 0x7f013c0afe40 2024-04-15T15:21:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:set pool 4 allow_ec_overwrites to true 2024-04-15T15:21:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f012006c920 msgr2=0x7f012006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f012006c920 0x7f012006ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0130036000 tx=0x7f0130039040 comp rx=0 tx=0).stop 2024-04-15T15:21:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c0afe40 msgr2=0x7f013c15baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:15.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c0afe40 0x7f013c15baa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f013400c760 tx=0x7f013400cc20 comp rx=0 tx=0).stop 2024-04-15T15:21:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 shutdown_connections 2024-04-15T15:21:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f013c0afe40 0x7f013c15baa0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f012006c920 0x7f012006ede0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f013c0d0a60 0x7f013c1f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 --2- 172.21.15.12:0/761169125 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f013c0b3b90 0x7f013c15e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 >> 172.21.15.12:0/761169125 conn(0x7f013c06bbc0 msgr2=0x7f013c0c55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.681+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 shutdown_connections 2024-04-15T15:21:15.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.682+0000 7f0142b59640 1 -- 172.21.15.12:0/761169125 wait complete. 2024-04-15T15:21:15.712 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-15T15:21:15.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.821+0000 7f47bc9bd640 1 Processor -- start 2024-04-15T15:21:15.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.821+0000 7f47bc9bd640 1 -- start start 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.822+0000 7f47bc9bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47b80d3270 0x7f47b80cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.822+0000 7f47bc9bd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b80b47a0 con 0x7f47b815e700 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.822+0000 7f47bc9bd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b80b4a80 con 0x7f47b80d2360 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.822+0000 7f47bc9bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f47b80b53b0 con 0x7f47b80d3270 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.822+0000 7f47b659c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f47b815e700 0x7f47b80b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58344/0 (socket says 172.21.15.12:58344) 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.822+0000 7f47b659c640 1 -- 172.21.15.12:0/1859907178 learned_addr learned my addr 172.21.15.12:0/1859907178 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.822+0000 7f47b6d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47b80d3270 0x7f47b80cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.823+0000 7f47b6d9d640 1 -- 172.21.15.12:0/1859907178 >> v1:172.21.15.55:6789/0 conn(0x7f47b80d2360 legacy=0x7f47b80d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.823+0000 7f47b6d9d640 1 -- 172.21.15.12:0/1859907178 >> v1:172.21.15.12:6789/0 conn(0x7f47b815e700 legacy=0x7f47b80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:15.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.823+0000 7f47b6d9d640 1 -- 172.21.15.12:0/1859907178 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b80b4d60 con 0x7f47b80d3270 2024-04-15T15:21:15.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.823+0000 7f47b6d9d640 1 --2- 172.21.15.12:0/1859907178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b80cf370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f47b80b5530 tx=0x7f47a802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d38aaca51c853d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47b559a640 1 -- 172.21.15.12:0/1859907178 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8039070 con 0x7f47b80d3270 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47b559a640 1 -- 172.21.15.12:0/1859907178 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a802cc50 con 0x7f47b80d3270 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47b559a640 1 -- 172.21.15.12:0/1859907178 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8034440 con 0x7f47b80d3270 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1859907178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 msgr2=0x7f47b80cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47bc9bd640 1 --2- 172.21.15.12:0/1859907178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b80cf370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f47b80b5530 tx=0x7f47a802bea0 comp rx=0 tx=0).stop 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1859907178 shutdown_connections 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47bc9bd640 1 --2- 172.21.15.12:0/1859907178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b80cf370 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1859907178 >> 172.21.15.12:0/1859907178 conn(0x7f47b808bf60 msgr2=0x7f47b80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1859907178 shutdown_connections 2024-04-15T15:21:15.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.824+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1859907178 wait complete. 2024-04-15T15:21:15.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.825+0000 7f47bc9bd640 1 Processor -- start 2024-04-15T15:21:15.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47bc9bd640 1 -- start start 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47bc9bd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b80d2360 0x7f47b8161a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47bc9bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b809e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47bc9bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47b815e700 0x7f47b80b95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47bc9bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f47b80d1d40 con 0x7f47b80d3270 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47bc9bd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f47b80b4450 con 0x7f47b80d2360 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47bc9bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f47b80d1ec0 con 0x7f47b815e700 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47b5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b809e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47b5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b809e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40938/0 (socket says 172.21.15.12:40938) 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.826+0000 7f47b5d9b640 1 -- 172.21.15.12:0/1996068436 learned_addr learned my addr 172.21.15.12:0/1996068436 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47b659c640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b80d2360 0x7f47b8161a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:15.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47b6d9d640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47b815e700 0x7f47b80b95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47b5d9b640 1 -- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47b815e700 msgr2=0x7f47b80b95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47b5d9b640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47b815e700 0x7f47b80b95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47b5d9b640 1 -- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b80d2360 msgr2=0x7f47b8161a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47b5d9b640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b80d2360 0x7f47b8161a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47b5d9b640 1 -- 172.21.15.12:0/1996068436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b815ec10 con 0x7f47b80d3270 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47b5d9b640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b809e270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f47a000c460 tx=0x7f47a000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47aeffd640 1 -- 172.21.15.12:0/1996068436 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0016020 con 0x7f47b80d3270 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47aeffd640 1 -- 172.21.15.12:0/1996068436 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a0012870 con 0x7f47b80d3270 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b815eef0 con 0x7f47b80d3270 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47aeffd640 1 -- 172.21.15.12:0/1996068436 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0012c60 con 0x7f47b80d3270 2024-04-15T15:21:15.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.827+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b815f4b0 con 0x7f47b80d3270 2024-04-15T15:21:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.828+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b815f990 con 0x7f47b80d3270 2024-04-15T15:21:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.829+0000 7f47aeffd640 1 -- 172.21.15.12:0/1996068436 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f47a0019020 con 0x7f47b80d3270 2024-04-15T15:21:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.829+0000 7f47aeffd640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f479806c8f0 0x7f479806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.829+0000 7f47aeffd640 1 -- 172.21.15.12:0/1996068436 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7f47a008b810 con 0x7f47b80d3270 2024-04-15T15:21:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.829+0000 7f47b659c640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f479806c8f0 0x7f479806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:15.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.830+0000 7f47b659c640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f479806c8f0 0x7f479806edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47b00010e0 tx=0x7f47b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:15.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:15.834+0000 7f47aeffd640 1 -- 172.21.15.12:0/1996068436 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47b815f990 con 0x7f47b80d3270 2024-04-15T15:21:16.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.062+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f47b80689e0 con 0x7f47b80d3270 2024-04-15T15:21:16.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.697+0000 7f47aeffd640 1 -- 172.21.15.12:0/1996068436 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7f47a0020070 con 0x7f47b80d3270 2024-04-15T15:21:16.699 INFO:teuthology.orchestra.run.smithi012.stderr:added data pool 4 to fsmap 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f479806c8f0 msgr2=0x7f479806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f479806c8f0 0x7f479806edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47b00010e0 tx=0x7f47b0009040 comp rx=0 tx=0).stop 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 msgr2=0x7f47b809e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b809e270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f47a000c460 tx=0x7f47a000c920 comp rx=0 tx=0).stop 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 shutdown_connections 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47b815e700 0x7f47b80b95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47b80d3270 0x7f47b809e270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f479806c8f0 0x7f479806edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 --2- 172.21.15.12:0/1996068436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b80d2360 0x7f47b8161a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 >> 172.21.15.12:0/1996068436 conn(0x7f47b808bf60 msgr2=0x7f47b80b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:16.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 shutdown_connections 2024-04-15T15:21:16.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.699+0000 7f47bc9bd640 1 -- 172.21.15.12:0/1996068436 wait complete. 2024-04-15T15:21:16.715 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:21:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.825+0000 7fa5a3374640 1 Processor -- start 2024-04-15T15:21:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.825+0000 7fa5a3374640 1 -- start start 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a3374640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa59c0d3480 0x7fa59c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a3374640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c0b6790 0x7fa59c0b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a3374640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59c0cff70 con 0x7fa59c15ea40 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a3374640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa59c0d0250 con 0x7fa59c0d3480 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a3374640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa59c0d0530 con 0x7fa59c0b6790 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a090f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa59c0d3480 0x7fa59c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a090f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa59c0d3480 0x7fa59c0b6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40946/0 (socket says 172.21.15.12:40946) 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a090f640 1 -- 172.21.15.12:0/3474415573 learned_addr learned my addr 172.21.15.12:0/3474415573 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a1911640 1 --2- 172.21.15.12:0/3474415573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c0b6790 0x7fa59c0b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a090f640 1 -- 172.21.15.12:0/3474415573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c0b6790 msgr2=0x7fa59c0b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a090f640 1 --2- 172.21.15.12:0/3474415573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c0b6790 0x7fa59c0b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a090f640 1 -- 172.21.15.12:0/3474415573 >> v1:172.21.15.55:6789/0 conn(0x7fa59c15ea40 legacy=0x7fa59c0b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a090f640 1 -- 172.21.15.12:0/3474415573 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59c0d0810 con 0x7fa59c0d3480 2024-04-15T15:21:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.826+0000 7fa5a1911640 1 --2- 172.21.15.12:0/3474415573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c0b6790 0x7fa59c0b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.827+0000 7fa5a090f640 1 --2- 172.21.15.12:0/3474415573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0d3480 0x7fa59c0b6250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009870 tx=0x7fa58c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7857935499b6979c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.827+0000 7fa593fff640 1 -- 172.21.15.12:0/3474415573 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c02f840 con 0x7fa59c0d3480 2024-04-15T15:21:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.827+0000 7fa593fff640 1 -- 172.21.15.12:0/3474415573 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa58c02f9e0 con 0x7fa59c0d3480 2024-04-15T15:21:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.827+0000 7fa593fff640 1 -- 172.21.15.12:0/3474415573 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c02fdb0 con 0x7fa59c0d3480 2024-04-15T15:21:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.828+0000 7fa5a3374640 1 -- 172.21.15.12:0/3474415573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0d3480 msgr2=0x7fa59c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.828+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3474415573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0d3480 0x7fa59c0b6250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009870 tx=0x7fa58c02ba20 comp rx=0 tx=0).stop 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.828+0000 7fa5a3374640 1 -- 172.21.15.12:0/3474415573 shutdown_connections 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.828+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3474415573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0d3480 0x7fa59c0b6250 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.828+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3474415573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c0b6790 0x7fa59c0b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.828+0000 7fa5a3374640 1 -- 172.21.15.12:0/3474415573 >> 172.21.15.12:0/3474415573 conn(0x7fa59c08bf60 msgr2=0x7fa59c0d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.828+0000 7fa5a3374640 1 -- 172.21.15.12:0/3474415573 shutdown_connections 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.828+0000 7fa5a3374640 1 -- 172.21.15.12:0/3474415573 wait complete. 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.829+0000 7fa5a3374640 1 Processor -- start 2024-04-15T15:21:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.829+0000 7fa5a3374640 1 -- start start 2024-04-15T15:21:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a3374640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0b6790 0x7fa59c154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a3374640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa59c0d3480 0x7fa59c0b9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a3374640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c15ea40 0x7fa59c153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a3374640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c0b3a40 con 0x7fa59c0b6790 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a3374640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c0b38c0 con 0x7fa59c0d3480 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a3374640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa59c15ee20 con 0x7fa59c15ea40 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0b6790 0x7fa59c154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0b6790 0x7fa59c154420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40962/0 (socket says 172.21.15.12:40962) 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 -- 172.21.15.12:0/3974406511 learned_addr learned my addr 172.21.15.12:0/3974406511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a090f640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa59c0d3480 0x7fa59c0b9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1911640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c15ea40 0x7fa59c153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 -- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c15ea40 msgr2=0x7fa59c153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c15ea40 0x7fa59c153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 -- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa59c0d3480 msgr2=0x7fa59c0b9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa59c0d3480 0x7fa59c0b9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 -- 172.21.15.12:0/3974406511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59c0b96a0 con 0x7fa59c0b6790 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.830+0000 7fa5a1110640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0b6790 0x7fa59c154420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa59800c3f0 tx=0x7fa59800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.831+0000 7fa591ffb640 1 -- 172.21.15.12:0/3974406511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5980052d0 con 0x7fa59c0b6790 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.831+0000 7fa591ffb640 1 -- 172.21.15.12:0/3974406511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa598005490 con 0x7fa59c0b6790 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.831+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59c0ba400 con 0x7fa59c0b6790 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.831+0000 7fa591ffb640 1 -- 172.21.15.12:0/3974406511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598007260 con 0x7fa59c0b6790 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.831+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59c0b8810 con 0x7fa59c0b6790 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.832+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa59c0690f0 con 0x7fa59c0b6790 2024-04-15T15:21:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.832+0000 7fa591ffb640 1 -- 172.21.15.12:0/3974406511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa598007400 con 0x7fa59c0b6790 2024-04-15T15:21:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.833+0000 7fa591ffb640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa58006c8f0 0x7fa58006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.833+0000 7fa591ffb640 1 -- 172.21.15.12:0/3974406511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa59808b2a0 con 0x7fa59c0b6790 2024-04-15T15:21:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.833+0000 7fa5a090f640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa58006c8f0 0x7fa58006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.834+0000 7fa5a090f640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa58006c8f0 0x7fa58006edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa58c036000 tx=0x7fa58c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:16.837+0000 7fa591ffb640 1 -- 172.21.15.12:0/3974406511 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa59c0690f0 con 0x7fa59c0b6790 2024-04-15T15:21:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.059+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa59c0689e0 con 0x7fa59c0b6790 2024-04-15T15:21:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.060+0000 7fa591ffb640 1 -- 172.21.15.12:0/3974406511 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5174 (secure 0 0 0) 0x7fa59808b0c0 con 0x7fa59c0b6790 2024-04-15T15:21:17.063 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:17.063 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:16.064128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:21:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-04-15T15:21:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa58006c8f0 msgr2=0x7fa58006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa58006c8f0 0x7fa58006edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa58c036000 tx=0x7fa58c008030 comp rx=0 tx=0).stop 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0b6790 msgr2=0x7fa59c154420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0b6790 0x7fa59c154420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa59800c3f0 tx=0x7fa59800c8b0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 shutdown_connections 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa59c15ea40 0x7fa59c153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa58006c8f0 0x7fa58006edb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa59c0d3480 0x7fa59c0b9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 --2- 172.21.15.12:0/3974406511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa59c0b6790 0x7fa59c154420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 >> 172.21.15.12:0/3974406511 conn(0x7fa59c08bf60 msgr2=0x7fa59c06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 shutdown_connections 2024-04-15T15:21:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.062+0000 7fa5a3374640 1 -- 172.21.15.12:0/3974406511 wait complete. 2024-04-15T15:21:17.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bbbd72640 1 Processor -- start 2024-04-15T15:21:17.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bbbd72640 1 -- start start 2024-04-15T15:21:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bbbd72640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3bb415e7a0 0x7f3bb40d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bbbd72640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d13f0 0x7f3bb4162370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bbbd72640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3bb40d1870 0x7f3bb41628b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bbbd72640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3bb40d0530 con 0x7f3bb40d1870 2024-04-15T15:21:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bbbd72640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3bb40cf490 con 0x7f3bb415e7a0 2024-04-15T15:21:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bbbd72640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb40cf770 con 0x7f3bb40d13f0 2024-04-15T15:21:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.177+0000 7f3bba30f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3bb40d1870 0x7f3bb41628b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bba30f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3bb40d1870 0x7f3bb41628b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40976/0 (socket says 172.21.15.12:40976) 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bba30f640 1 -- 172.21.15.12:0/2871545313 learned_addr learned my addr 172.21.15.12:0/2871545313 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bb9b0e640 1 --2- 172.21.15.12:0/2871545313 >> v2:172.21.15.55:3300/0 conn(0x7f3bb415e7a0 0x7f3bb40d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bb930d640 1 --2- 172.21.15.12:0/2871545313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d13f0 0x7f3bb4162370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bba30f640 1 -- 172.21.15.12:0/2871545313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d13f0 msgr2=0x7f3bb4162370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bba30f640 1 --2- 172.21.15.12:0/2871545313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d13f0 0x7f3bb4162370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bba30f640 1 -- 172.21.15.12:0/2871545313 >> v2:172.21.15.55:3300/0 conn(0x7f3bb415e7a0 msgr2=0x7f3bb40d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bba30f640 1 --2- 172.21.15.12:0/2871545313 >> v2:172.21.15.55:3300/0 conn(0x7f3bb415e7a0 0x7f3bb40d0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bba30f640 1 -- 172.21.15.12:0/2871545313 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb40cfa50 con 0x7f3bb40d1870 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.178+0000 7f3bb9b0e640 1 --2- 172.21.15.12:0/2871545313 >> v2:172.21.15.55:3300/0 conn(0x7f3bb415e7a0 0x7f3bb40d0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bba30f640 1 --2- 172.21.15.12:0/2871545313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb40d1870 0x7f3bb41628b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40b7110 tx=0x7f3bac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6467f66c3965787 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bb8b0c640 1 -- 172.21.15.12:0/2871545313 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac02c9e0 con 0x7f3bb40d1870 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bb8b0c640 1 -- 172.21.15.12:0/2871545313 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bac02cb80 con 0x7f3bb40d1870 2024-04-15T15:21:17.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bb8b0c640 1 -- 172.21.15.12:0/2871545313 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac0343f0 con 0x7f3bb40d1870 2024-04-15T15:21:17.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 -- 172.21.15.12:0/2871545313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb40d1870 msgr2=0x7f3bb41628b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/2871545313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb40d1870 0x7f3bb41628b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40b7110 tx=0x7f3bac02be40 comp rx=0 tx=0).stop 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 -- 172.21.15.12:0/2871545313 shutdown_connections 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/2871545313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb40d1870 0x7f3bb41628b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/2871545313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d13f0 0x7f3bb4162370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/2871545313 >> v2:172.21.15.55:3300/0 conn(0x7f3bb415e7a0 0x7f3bb40d0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 -- 172.21.15.12:0/2871545313 >> 172.21.15.12:0/2871545313 conn(0x7f3bb408bf60 msgr2=0x7f3bb40d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 -- 172.21.15.12:0/2871545313 shutdown_connections 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.179+0000 7f3bbbd72640 1 -- 172.21.15.12:0/2871545313 wait complete. 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bbbd72640 1 Processor -- start 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bbbd72640 1 -- start start 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bbbd72640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb40d13f0 0x7f3bb4161320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bbbd72640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d1870 0x7f3bb40a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bbbd72640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb415e7a0 0x7f3bb414e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bbbd72640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb415eb80 con 0x7f3bb415e7a0 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bbbd72640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb40cf180 con 0x7f3bb40d13f0 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bbbd72640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb4162df0 con 0x7f3bb40d1870 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bba30f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb415e7a0 0x7f3bb414e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bba30f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb415e7a0 0x7f3bb414e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40990/0 (socket says 172.21.15.12:40990) 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.181+0000 7f3bba30f640 1 -- 172.21.15.12:0/3544278231 learned_addr learned my addr 172.21.15.12:0/3544278231 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bb9b0e640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb40d13f0 0x7f3bb4161320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bb930d640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d1870 0x7f3bb40a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bba30f640 1 -- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d1870 msgr2=0x7f3bb40a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bba30f640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d1870 0x7f3bb40a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bba30f640 1 -- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb40d13f0 msgr2=0x7f3bb4161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bba30f640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb40d13f0 0x7f3bb4161320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bba30f640 1 -- 172.21.15.12:0/3544278231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb40a2000 con 0x7f3bb415e7a0 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bb9b0e640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb40d13f0 0x7f3bb4161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bba30f640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb415e7a0 0x7f3bb414e9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3bac036040 tx=0x7f3bac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3baa7fc640 1 -- 172.21.15.12:0/3544278231 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac043020 con 0x7f3bb415e7a0 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3baa7fc640 1 -- 172.21.15.12:0/3544278231 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bac03e450 con 0x7f3bb415e7a0 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3baa7fc640 1 -- 172.21.15.12:0/3544278231 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac03e820 con 0x7f3bb415e7a0 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb40a1480 con 0x7f3bb415e7a0 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.182+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb40a1a40 con 0x7f3bb415e7a0 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.183+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bb40689e0 con 0x7f3bb415e7a0 2024-04-15T15:21:17.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.183+0000 7f3baa7fc640 1 -- 172.21.15.12:0/3544278231 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3bac046020 con 0x7f3bb415e7a0 2024-04-15T15:21:17.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.184+0000 7f3baa7fc640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b9806c8f0 0x7f3b9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.184+0000 7f3baa7fc640 1 -- 172.21.15.12:0/3544278231 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3bac0ae790 con 0x7f3bb415e7a0 2024-04-15T15:21:17.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.184+0000 7f3bb9b0e640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b9806c8f0 0x7f3b9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.184+0000 7f3bb9b0e640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b9806c8f0 0x7f3b9806edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000a990 tx=0x7f3bb0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.189+0000 7f3baa7fc640 1 -- 172.21.15.12:0/3544278231 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3bac04d030 con 0x7f3bb415e7a0 2024-04-15T15:21:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.372+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3bb40cc100 con 0x7f3bb415e7a0 2024-04-15T15:21:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.373+0000 7f3baa7fc640 1 -- 172.21.15.12:0/3544278231 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f3bac07c910 con 0x7f3bb415e7a0 2024-04-15T15:21:17.376 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:17.376 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b9806c8f0 msgr2=0x7f3b9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b9806c8f0 0x7f3b9806edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000a990 tx=0x7f3bb0008040 comp rx=0 tx=0).stop 2024-04-15T15:21:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb415e7a0 msgr2=0x7f3bb414e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb415e7a0 0x7f3bb414e9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3bac036040 tx=0x7f3bac004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 shutdown_connections 2024-04-15T15:21:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bb415e7a0 0x7f3bb414e9d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bb40d1870 0x7f3bb40a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3b9806c8f0 0x7f3b9806edb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 --2- 172.21.15.12:0/3544278231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb40d13f0 0x7f3bb4161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 >> 172.21.15.12:0/3544278231 conn(0x7f3bb408bf60 msgr2=0x7f3bb415d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 shutdown_connections 2024-04-15T15:21:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.375+0000 7f3bbbd72640 1 -- 172.21.15.12:0/3544278231 wait complete. 2024-04-15T15:21:17.389 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:17.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.491+0000 7f14eeaf2640 1 Processor -- start 2024-04-15T15:21:17.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.491+0000 7f14eeaf2640 1 -- start start 2024-04-15T15:21:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14eeaf2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f14e815e790 0x7f14e80d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14eeaf2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80cfd20 0x7f14e80d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14eeaf2640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e80d2480 con 0x7f14e80d0f00 2024-04-15T15:21:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14eeaf2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f14e80d2db0 con 0x7f14e815e790 2024-04-15T15:21:17.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14eeaf2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f14e80d2760 con 0x7f14e80cfd20 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14ec88e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f14e815e790 0x7f14e80d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14e7fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80cfd20 0x7f14e80d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14ed08f640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f14e80d0f00 0x7f14e80b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:35652/0 (socket says 172.21.15.12:35652) 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14ed08f640 1 -- 172.21.15.12:0/4230327544 learned_addr learned my addr 172.21.15.12:0/4230327544 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14ec88e640 1 -- 172.21.15.12:0/4230327544 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80cfd20 msgr2=0x7f14e80d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14ec88e640 1 --2- 172.21.15.12:0/4230327544 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80cfd20 0x7f14e80d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14ec88e640 1 -- 172.21.15.12:0/4230327544 >> v1:172.21.15.55:6789/0 conn(0x7f14e80d0f00 legacy=0x7f14e80b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14ec88e640 1 -- 172.21.15.12:0/4230327544 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e80d2a40 con 0x7f14e815e790 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14e7fff640 1 --2- 172.21.15.12:0/4230327544 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80cfd20 0x7f14e80d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:21:17.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14ec88e640 1 --2- 172.21.15.12:0/4230327544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 0x7f14e80d3240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14e00097a0 tx=0x7f14e002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a02c6bcf880509f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14e77fe640 1 -- 172.21.15.12:0/4230327544 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0035020 con 0x7f14e815e790 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.493+0000 7f14e77fe640 1 -- 172.21.15.12:0/4230327544 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e0031830 con 0x7f14e815e790 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14e77fe640 1 -- 172.21.15.12:0/4230327544 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0031c00 con 0x7f14e815e790 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14eeaf2640 1 -- 172.21.15.12:0/4230327544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 msgr2=0x7f14e80d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/4230327544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 0x7f14e80d3240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14e00097a0 tx=0x7f14e002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14eeaf2640 1 -- 172.21.15.12:0/4230327544 shutdown_connections 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/4230327544 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80cfd20 0x7f14e80d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/4230327544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 0x7f14e80d3240 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14eeaf2640 1 -- 172.21.15.12:0/4230327544 >> 172.21.15.12:0/4230327544 conn(0x7f14e808bf60 msgr2=0x7f14e80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14eeaf2640 1 -- 172.21.15.12:0/4230327544 shutdown_connections 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.494+0000 7f14eeaf2640 1 -- 172.21.15.12:0/4230327544 wait complete. 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.495+0000 7f14eeaf2640 1 Processor -- start 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.495+0000 7f14eeaf2640 1 -- start start 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14eeaf2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e80cfd20 0x7f14e80ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14eeaf2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80d0f00 0x7f14e80c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14eeaf2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 0x7f14e81e9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14eeaf2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f14e815ecf0 con 0x7f14e80cfd20 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14eeaf2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f14e815eb70 con 0x7f14e815e790 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14eeaf2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f14e80d13b0 con 0x7f14e80d0f00 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ec88e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e80cfd20 0x7f14e80ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ec88e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e80cfd20 0x7f14e80ca790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40992/0 (socket says 172.21.15.12:40992) 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ec88e640 1 -- 172.21.15.12:0/1781984138 learned_addr learned my addr 172.21.15.12:0/1781984138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ed08f640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 0x7f14e81e9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14e7fff640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80d0f00 0x7f14e80c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ec88e640 1 -- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80d0f00 msgr2=0x7f14e80c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ec88e640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80d0f00 0x7f14e80c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ec88e640 1 -- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 msgr2=0x7f14e81e9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ec88e640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 0x7f14e81e9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14ec88e640 1 -- 172.21.15.12:0/1781984138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e80c8f90 con 0x7f14e80cfd20 2024-04-15T15:21:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.496+0000 7f14e7fff640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80d0f00 0x7f14e80c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.497+0000 7f14ec88e640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e80cfd20 0x7f14e80ca790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f14e0034040 tx=0x7f14e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.497+0000 7f14e57fa640 1 -- 172.21.15.12:0/1781984138 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0040070 con 0x7f14e80cfd20 2024-04-15T15:21:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.497+0000 7f14e57fa640 1 -- 172.21.15.12:0/1781984138 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e003a910 con 0x7f14e80cfd20 2024-04-15T15:21:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.497+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e80c9270 con 0x7f14e80cfd20 2024-04-15T15:21:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.497+0000 7f14e57fa640 1 -- 172.21.15.12:0/1781984138 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0035040 con 0x7f14e80cfd20 2024-04-15T15:21:17.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.497+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e81f5a50 con 0x7f14e80cfd20 2024-04-15T15:21:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.498+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e80c9790 con 0x7f14e80cfd20 2024-04-15T15:21:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.498+0000 7f14e57fa640 1 -- 172.21.15.12:0/1781984138 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14e0038020 con 0x7f14e80cfd20 2024-04-15T15:21:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.499+0000 7f14e57fa640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f14c806c8f0 0x7f14c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.499+0000 7f14e57fa640 1 -- 172.21.15.12:0/1781984138 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f14e00ad7e0 con 0x7f14e80cfd20 2024-04-15T15:21:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.499+0000 7f14e7fff640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f14c806c8f0 0x7f14c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.499+0000 7f14e7fff640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f14c806c8f0 0x7f14c806edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f14e81e98d0 tx=0x7f14d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.504+0000 7f14e57fa640 1 -- 172.21.15.12:0/1781984138 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14e80c9790 con 0x7f14e80cfd20 2024-04-15T15:21:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.688+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14e80689e0 con 0x7f14e80cfd20 2024-04-15T15:21:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.690+0000 7f14e57fa640 1 -- 172.21.15.12:0/1781984138 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f14e0031800 con 0x7f14e80cfd20 2024-04-15T15:21:17.692 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:17.692 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.691+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f14c806c8f0 msgr2=0x7f14c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.691+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f14c806c8f0 0x7f14c806edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f14e81e98d0 tx=0x7f14d0008040 comp rx=0 tx=0).stop 2024-04-15T15:21:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.691+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e80cfd20 msgr2=0x7f14e80ca790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.691+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e80cfd20 0x7f14e80ca790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f14e0034040 tx=0x7f14e00046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.692+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 shutdown_connections 2024-04-15T15:21:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.692+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f14c806c8f0 0x7f14c806edb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.692+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14e815e790 0x7f14e81e9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.692+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f14e80d0f00 0x7f14e80c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.692+0000 7f14eeaf2640 1 --2- 172.21.15.12:0/1781984138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14e80cfd20 0x7f14e80ca790 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.692+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 >> 172.21.15.12:0/1781984138 conn(0x7f14e808bf60 msgr2=0x7f14e80caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.692+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 shutdown_connections 2024-04-15T15:21:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.692+0000 7f14eeaf2640 1 -- 172.21.15.12:0/1781984138 wait complete. 2024-04-15T15:21:17.719 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:21:17.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4718f52640 1 Processor -- start 2024-04-15T15:21:17.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4718f52640 1 -- start start 2024-04-15T15:21:17.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4718f52640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f471415ea40 0x7f47140b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4718f52640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 0x7f47140b6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4718f52640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47140cff70 con 0x7f47140b65c0 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4718f52640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f47140b6a10 con 0x7f471415ea40 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4718f52640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f47140d0250 con 0x7f471415ee20 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4712d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f47140b65c0 0x7f47140d2470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58356/0 (socket says 172.21.15.12:58356) 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4712d9d640 1 -- 172.21.15.12:0/153974238 learned_addr learned my addr 172.21.15.12:0/153974238 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f471259c640 1 --2- 172.21.15.12:0/153974238 >> v2:172.21.15.55:3300/0 conn(0x7f471415ea40 0x7f47140b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f470bfff640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 0x7f47140b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2351285085 0 0) 0x7f47140cff70 con 0x7f47140b65c0 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4700003520 con 0x7f47140b65c0 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4064319941 0 0) 0x7f4700003520 con 0x7f47140b65c0 2024-04-15T15:21:17.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 msgr2=0x7f47140b6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 0x7f47140b6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 >> v2:172.21.15.55:3300/0 conn(0x7f471415ea40 msgr2=0x7f47140b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 --2- 172.21.15.12:0/153974238 >> v2:172.21.15.55:3300/0 conn(0x7f471415ea40 0x7f47140b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47140d0530 con 0x7f47140b65c0 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f470c0019f0 con 0x7f47140b65c0 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f470bfff640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 0x7f47140b6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 >> v1:172.21.15.12:6789/0 conn(0x7f47140b65c0 legacy=0x7f47140d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 0x7f47000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4700003ef0 0x7f4700004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f47000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f47140b6a10 con 0x7f4700004d40 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f47140d0250 con 0x7f4700003ef0 2024-04-15T15:21:17.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f470000b9c0 con 0x7f471415ee20 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f4712d9d640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f47000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f471259c640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 0x7f47000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.823+0000 7f470bfff640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4700003ef0 0x7f4700004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4712d9d640 1 -- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 msgr2=0x7f47000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4712d9d640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471415ee20 0x7f47000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4712d9d640 1 -- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4700003ef0 msgr2=0x7f4700004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4712d9d640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4700003ef0 0x7f4700004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4712d9d640 1 -- 172.21.15.12:0/153974238 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4700003520 con 0x7f4700004d40 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4712d9d640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f47000051c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f470c00a950 tx=0x7f4700005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dcc37575d86626 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c00a4c0 con 0x7f4700004d40 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470c034650 con 0x7f4700004d40 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.824+0000 7f4711d9b640 1 -- 172.21.15.12:0/153974238 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c034a80 con 0x7f4700004d40 2024-04-15T15:21:17.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.825+0000 7f4718f52640 1 -- 172.21.15.12:0/153974238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 msgr2=0x7f47000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.825+0000 7f4718f52640 1 --2- 172.21.15.12:0/153974238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f47000051c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f470c00a950 tx=0x7f4700005830 comp rx=0 tx=0).stop 2024-04-15T15:21:17.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.825+0000 7f4712d9d640 1 -- 172.21.15.12:0/153974238 reap_dead start 2024-04-15T15:21:17.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.825+0000 7f4718f52640 1 -- 172.21.15.12:0/153974238 shutdown_connections 2024-04-15T15:21:17.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.825+0000 7f4718f52640 1 -- 172.21.15.12:0/153974238 >> 172.21.15.12:0/153974238 conn(0x7f471408bf60 msgr2=0x7f47140d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:17.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.825+0000 7f4718f52640 1 -- 172.21.15.12:0/153974238 shutdown_connections 2024-04-15T15:21:17.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.825+0000 7f4718f52640 1 -- 172.21.15.12:0/153974238 wait complete. 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.826+0000 7f4718f52640 1 Processor -- start 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.826+0000 7f4718f52640 1 -- start start 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f4718f52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f4714152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f4718f52640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4714156470 0x7f471415fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f4718f52640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47140d0c70 0x7f47140d1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f4718f52640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f47140b38c0 con 0x7f4700004d40 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f4718f52640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f47140d0af0 con 0x7f4714156470 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f4718f52640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f47140b3a40 con 0x7f47140d0c70 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f471259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f4714152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f471259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f4714152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41024/0 (socket says 172.21.15.12:41024) 2024-04-15T15:21:17.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f471259c640 1 -- 172.21.15.12:0/2328369361 learned_addr learned my addr 172.21.15.12:0/2328369361 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f470bfff640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4714156470 0x7f471415fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f471259c640 1 -- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47140d0c70 msgr2=0x7f47140d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f4712d9d640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47140d0c70 0x7f47140d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f471259c640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47140d0c70 0x7f47140d1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f471259c640 1 -- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4714156470 msgr2=0x7f471415fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f471259c640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4714156470 0x7f471415fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.827+0000 7f471259c640 1 -- 172.21.15.12:0/2328369361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4714161e40 con 0x7f4700004d40 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.828+0000 7f470bfff640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4714156470 0x7f471415fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.828+0000 7f471259c640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f4714152b50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4704011aa0 tx=0x7f4704011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.828+0000 7f470affd640 1 -- 172.21.15.12:0/2328369361 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470400fb80 con 0x7f4700004d40 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.828+0000 7f470affd640 1 -- 172.21.15.12:0/2328369361 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470400fd20 con 0x7f4700004d40 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.828+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47141553d0 con 0x7f4700004d40 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.828+0000 7f470affd640 1 -- 172.21.15.12:0/2328369361 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704014660 con 0x7f4700004d40 2024-04-15T15:21:17.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.828+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4714155990 con 0x7f4700004d40 2024-04-15T15:21:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.829+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47140689e0 con 0x7f4700004d40 2024-04-15T15:21:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.829+0000 7f470affd640 1 -- 172.21.15.12:0/2328369361 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4704012040 con 0x7f4700004d40 2024-04-15T15:21:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.830+0000 7f470affd640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f406c920 0x7f46f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.830+0000 7f470affd640 1 -- 172.21.15.12:0/2328369361 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f470408c520 con 0x7f4700004d40 2024-04-15T15:21:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.830+0000 7f470bfff640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f406c920 0x7f46f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.830+0000 7f470bfff640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f406c920 0x7f46f406ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4714156a60 tx=0x7f46fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:17.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:17.835+0000 7f470affd640 1 -- 172.21.15.12:0/2328369361 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4704019030 con 0x7f4700004d40 2024-04-15T15:21:18.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.054+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47140b58f0 con 0x7f4700004d40 2024-04-15T15:21:18.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.055+0000 7f470affd640 1 -- 172.21.15.12:0/2328369361 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5174 (secure 0 0 0) 0x7f470405a6a0 con 0x7f4700004d40 2024-04-15T15:21:18.057 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:18.057 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:16.064128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:21:18.058 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-04-15T15:21:18.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f406c920 msgr2=0x7f46f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f406c920 0x7f46f406ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4714156a60 tx=0x7f46fc008040 comp rx=0 tx=0).stop 2024-04-15T15:21:18.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 msgr2=0x7f4714152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f4714152b50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4704011aa0 tx=0x7f4704011f60 comp rx=0 tx=0).stop 2024-04-15T15:21:18.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 shutdown_connections 2024-04-15T15:21:18.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47140d0c70 0x7f47140d1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f406c920 0x7f46f406ede0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4714156470 0x7f471415fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 --2- 172.21.15.12:0/2328369361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4700004d40 0x7f4714152b50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 >> 172.21.15.12:0/2328369361 conn(0x7f471408bf60 msgr2=0x7f471406e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 shutdown_connections 2024-04-15T15:21:18.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.057+0000 7f4718f52640 1 -- 172.21.15.12:0/2328369361 wait complete. 2024-04-15T15:21:18.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.172+0000 7f5d78a63640 1 Processor -- start 2024-04-15T15:21:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.172+0000 7f5d78a63640 1 -- start start 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d78a63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d740b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d78a63640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d740d2480 con 0x7f5d7415e790 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d78a63640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d740d2760 con 0x7f5d740d3240 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d78a63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5d740d2db0 con 0x7f5d740d1610 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d7259c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5d7415e790 0x7f5d740cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58368/0 (socket says 172.21.15.12:58368) 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d7259c640 1 -- 172.21.15.12:0/2070530409 learned_addr learned my addr 172.21.15.12:0/2070530409 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d72d9d640 1 --2- 172.21.15.12:0/2070530409 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d740b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d71d9b640 1 -- 172.21.15.12:0/2070530409 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864455422 0 0) 0x7f5d740d2480 con 0x7f5d7415e790 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d71d9b640 1 -- 172.21.15.12:0/2070530409 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d60003520 con 0x7f5d7415e790 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d71d9b640 1 -- 172.21.15.12:0/2070530409 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3826779614 0 0) 0x7f5d60003520 con 0x7f5d7415e790 2024-04-15T15:21:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d72d9d640 1 -- 172.21.15.12:0/2070530409 >> v1:172.21.15.55:6789/0 conn(0x7f5d740d3240 legacy=0x7f5d740d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d72d9d640 1 -- 172.21.15.12:0/2070530409 >> v1:172.21.15.12:6789/0 conn(0x7f5d7415e790 legacy=0x7f5d740cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.173+0000 7f5d72d9d640 1 -- 172.21.15.12:0/2070530409 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d740d2a40 con 0x7f5d740d1610 2024-04-15T15:21:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.174+0000 7f5d72d9d640 1 --2- 172.21.15.12:0/2070530409 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d740b74f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009dc0 tx=0x7f5d6c02bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=552ddf8818a69348 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.174+0000 7f5d71d9b640 1 -- 172.21.15.12:0/2070530409 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c039070 con 0x7f5d740d1610 2024-04-15T15:21:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.174+0000 7f5d71d9b640 1 -- 172.21.15.12:0/2070530409 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d6c02c770 con 0x7f5d740d1610 2024-04-15T15:21:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.174+0000 7f5d71d9b640 1 -- 172.21.15.12:0/2070530409 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64001690 con 0x7f5d740d1610 2024-04-15T15:21:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.175+0000 7f5d78a63640 1 -- 172.21.15.12:0/2070530409 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 msgr2=0x7f5d740b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.175+0000 7f5d78a63640 1 --2- 172.21.15.12:0/2070530409 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d740b74f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009dc0 tx=0x7f5d6c02bc40 comp rx=0 tx=0).stop 2024-04-15T15:21:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.175+0000 7f5d78a63640 1 -- 172.21.15.12:0/2070530409 shutdown_connections 2024-04-15T15:21:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.175+0000 7f5d78a63640 1 --2- 172.21.15.12:0/2070530409 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d740b74f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.175+0000 7f5d78a63640 1 -- 172.21.15.12:0/2070530409 >> 172.21.15.12:0/2070530409 conn(0x7f5d7408bf60 msgr2=0x7f5d740d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.175+0000 7f5d78a63640 1 -- 172.21.15.12:0/2070530409 shutdown_connections 2024-04-15T15:21:18.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.175+0000 7f5d78a63640 1 -- 172.21.15.12:0/2070530409 wait complete. 2024-04-15T15:21:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.176+0000 7f5d78a63640 1 Processor -- start 2024-04-15T15:21:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.176+0000 7f5d78a63640 1 -- start start 2024-04-15T15:21:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d78a63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d741f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d78a63640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d740d3240 0x7f5d74158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d78a63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d7415e790 0x7f5d74158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d78a63640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d740d5180 con 0x7f5d7415e790 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d78a63640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5d740d0430 con 0x7f5d740d3240 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d78a63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5d740d5300 con 0x7f5d740d1610 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d72d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d7415e790 0x7f5d74158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d72d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d7415e790 0x7f5d74158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41034/0 (socket says 172.21.15.12:41034) 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d72d9d640 1 -- 172.21.15.12:0/2686514306 learned_addr learned my addr 172.21.15.12:0/2686514306 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d7259c640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d741f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.177+0000 7f5d69d9b640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d740d3240 0x7f5d74158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d72d9d640 1 -- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 msgr2=0x7f5d741f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d72d9d640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d741f3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d72d9d640 1 -- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d740d3240 msgr2=0x7f5d74158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d72d9d640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d740d3240 0x7f5d74158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d72d9d640 1 -- 172.21.15.12:0/2686514306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d741f46e0 con 0x7f5d7415e790 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d7259c640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d741f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d69d9b640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d740d3240 0x7f5d74158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d72d9d640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d7415e790 0x7f5d74158eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c037040 tx=0x7f5d6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d6b7fe640 1 -- 172.21.15.12:0/2686514306 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c044020 con 0x7f5d7415e790 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d6b7fe640 1 -- 172.21.15.12:0/2686514306 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d6c034a90 con 0x7f5d7415e790 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d741f49c0 con 0x7f5d7415e790 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d6b7fe640 1 -- 172.21.15.12:0/2686514306 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c039040 con 0x7f5d7415e790 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.178+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d741f4f80 con 0x7f5d7415e790 2024-04-15T15:21:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.179+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d740689e0 con 0x7f5d7415e790 2024-04-15T15:21:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.179+0000 7f5d6b7fe640 1 -- 172.21.15.12:0/2686514306 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d6c047020 con 0x7f5d7415e790 2024-04-15T15:21:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.180+0000 7f5d6b7fe640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5d5406c8f0 0x7f5d5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.180+0000 7f5d6b7fe640 1 -- 172.21.15.12:0/2686514306 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5d6c0ae520 con 0x7f5d7415e790 2024-04-15T15:21:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.180+0000 7f5d7259c640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5d5406c8f0 0x7f5d5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.181+0000 7f5d7259c640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5d5406c8f0 0x7f5d5406edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5d740ced50 tx=0x7f5d64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.185+0000 7f5d6b7fe640 1 -- 172.21.15.12:0/2686514306 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d6c04e030 con 0x7f5d7415e790 2024-04-15T15:21:18.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.374+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d740d1ce0 con 0x7f5d7415e790 2024-04-15T15:21:18.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.375+0000 7f5d6b7fe640 1 -- 172.21.15.12:0/2686514306 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f5d6c0333a0 con 0x7f5d7415e790 2024-04-15T15:21:18.378 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:18.378 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5d5406c8f0 msgr2=0x7f5d5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5d5406c8f0 0x7f5d5406edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5d740ced50 tx=0x7f5d64009040 comp rx=0 tx=0).stop 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d7415e790 msgr2=0x7f5d74158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d7415e790 0x7f5d74158eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c037040 tx=0x7f5d6c004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 shutdown_connections 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d7415e790 0x7f5d74158eb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5d5406c8f0 0x7f5d5406edb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d740d3240 0x7f5d74158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 --2- 172.21.15.12:0/2686514306 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5d740d1610 0x7f5d741f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 >> 172.21.15.12:0/2686514306 conn(0x7f5d7408bf60 msgr2=0x7f5d740d4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 shutdown_connections 2024-04-15T15:21:18.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.378+0000 7f5d78a63640 1 -- 172.21.15.12:0/2686514306 wait complete. 2024-04-15T15:21:18.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:21:11.000401+0000', 'modified': '2024-04-15T15:21:16.064128+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.55:6837/1379664958', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 1379664958}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 1379664958}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4290, 'qdb_cluster': [4290]} 2024-04-15T15:21:18.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:21:18.794 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-15T15:21:18.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.900+0000 7f83e0e37640 1 Processor -- start 2024-04-15T15:21:18.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.901+0000 7f83e0e37640 1 -- start start 2024-04-15T15:21:18.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.901+0000 7f83e0e37640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc15ebb0 0x7f83dc06cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.901+0000 7f83e0e37640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f83dc09b780 0x7f83dc0b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.901+0000 7f83e0e37640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83dc0cccb0 con 0x7f83dc09a9d0 2024-04-15T15:21:18.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.901+0000 7f83e0e37640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f83dc0cd8c0 con 0x7f83dc09b780 2024-04-15T15:21:18.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.901+0000 7f83e0e37640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f83dc0ced60 con 0x7f83dc15ebb0 2024-04-15T15:21:18.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83dad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f83dc09b780 0x7f83dc0b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83dad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f83dc09b780 0x7f83dc0b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41042/0 (socket says 172.21.15.12:41042) 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83dad9d640 1 -- 172.21.15.12:0/1525506710 learned_addr learned my addr 172.21.15.12:0/1525506710 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/1525506710 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc15ebb0 0x7f83dc06cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83dad9d640 1 -- 172.21.15.12:0/1525506710 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc15ebb0 msgr2=0x7f83dc06cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83dad9d640 1 --2- 172.21.15.12:0/1525506710 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc15ebb0 0x7f83dc06cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83dad9d640 1 -- 172.21.15.12:0/1525506710 >> v1:172.21.15.55:6789/0 conn(0x7f83dc09a9d0 legacy=0x7f83dc09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83dad9d640 1 -- 172.21.15.12:0/1525506710 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc0ccf90 con 0x7f83dc09b780 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.902+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/1525506710 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc15ebb0 0x7f83dc06cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83dad9d640 1 --2- 172.21.15.12:0/1525506710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc09b780 0x7f83dc0b76f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009e50 tx=0x7f83d402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9115bb368f17619e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83d959a640 1 -- 172.21.15.12:0/1525506710 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d402cb00 con 0x7f83dc09b780 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83d959a640 1 -- 172.21.15.12:0/1525506710 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d402cca0 con 0x7f83dc09b780 2024-04-15T15:21:18.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83d959a640 1 -- 172.21.15.12:0/1525506710 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4034610 con 0x7f83dc09b780 2024-04-15T15:21:18.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83e0e37640 1 -- 172.21.15.12:0/1525506710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc09b780 msgr2=0x7f83dc0b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1525506710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc09b780 0x7f83dc0b76f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009e50 tx=0x7f83d402bf60 comp rx=0 tx=0).stop 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83e0e37640 1 -- 172.21.15.12:0/1525506710 shutdown_connections 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1525506710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc09b780 0x7f83dc0b76f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1525506710 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc15ebb0 0x7f83dc06cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83e0e37640 1 -- 172.21.15.12:0/1525506710 >> 172.21.15.12:0/1525506710 conn(0x7f83dc0b3fb0 msgr2=0x7f83dc0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83e0e37640 1 -- 172.21.15.12:0/1525506710 shutdown_connections 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.903+0000 7f83e0e37640 1 -- 172.21.15.12:0/1525506710 wait complete. 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.905+0000 7f83e0e37640 1 Processor -- start 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.905+0000 7f83e0e37640 1 -- start start 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.905+0000 7f83e0e37640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83dc09a9d0 0x7f83dc1f80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.905+0000 7f83e0e37640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc09b780 0x7f83dc1f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.905+0000 7f83e0e37640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc1609e0 0x7f83dc0b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.905+0000 7f83e0e37640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc06d240 con 0x7f83dc1609e0 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.905+0000 7f83e0e37640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc0cd550 con 0x7f83dc09a9d0 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.905+0000 7f83e0e37640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f83dc0b5310 con 0x7f83dc09b780 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc1609e0 0x7f83dc0b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc1609e0 0x7f83dc0b69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41046/0 (socket says 172.21.15.12:41046) 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 -- 172.21.15.12:0/1178793772 learned_addr learned my addr 172.21.15.12:0/1178793772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83da59c640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83dc09a9d0 0x7f83dc1f80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc09b780 0x7f83dc1f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 -- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc09b780 msgr2=0x7f83dc1f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc09b780 0x7f83dc1f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 -- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83dc09a9d0 msgr2=0x7f83dc1f80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83dc09a9d0 0x7f83dc1f80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 -- 172.21.15.12:0/1178793772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc1f44c0 con 0x7f83dc1609e0 2024-04-15T15:21:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83dad9d640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc1609e0 0x7f83dc0b69d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009f70 tx=0x7f83d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83caffd640 1 -- 172.21.15.12:0/1178793772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4043020 con 0x7f83dc1609e0 2024-04-15T15:21:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83caffd640 1 -- 172.21.15.12:0/1178793772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d402ce50 con 0x7f83dc1609e0 2024-04-15T15:21:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83e0e37640 1 -- 172.21.15.12:0/1178793772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83dc1f47a0 con 0x7f83dc1609e0 2024-04-15T15:21:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83caffd640 1 -- 172.21.15.12:0/1178793772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d403e690 con 0x7f83dc1609e0 2024-04-15T15:21:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.906+0000 7f83e0e37640 1 -- 172.21.15.12:0/1178793772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83dc1f4d60 con 0x7f83dc1609e0 2024-04-15T15:21:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.908+0000 7f83caffd640 1 -- 172.21.15.12:0/1178793772 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f83d4046020 con 0x7f83dc1609e0 2024-04-15T15:21:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.908+0000 7f83caffd640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f83bc06c8f0 0x7f83bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.909+0000 7f83caffd640 1 -- 172.21.15.12:0/1178793772 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f83d40ae7e0 con 0x7f83dc1609e0 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.909+0000 7f83da59c640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f83bc06c8f0 0x7f83bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.909+0000 7f83da59c640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f83bc06c8f0 0x7f83bc06edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f83cc001640 tx=0x7f83cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.909+0000 7f83e3a19740 1 Processor -- start 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.909+0000 7f83e3a19740 1 -- start start 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f8e10 0x55cb649cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 0x55cb649fbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649cc2f0 0x55cb649fc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83e3a19740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55cb649369c0 con 0x55cb649cc2f0 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83e3a19740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55cb648dd650 con 0x55cb649f8e10 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83e3a19740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55cb64932980 con 0x55cb649f91f0 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83dad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649cc2f0 0x55cb649fc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83dad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649cc2f0 0x55cb649fc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41058/0 (socket says 172.21.15.12:41058) 2024-04-15T15:21:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83dad9d640 1 -- 172.21.15.12:0/1936221469 learned_addr learned my addr 172.21.15.12:0/1936221469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83da59c640 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f8e10 0x55cb649cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.910+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 0x55cb649fbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83dad9d640 1 -- 172.21.15.12:0/1936221469 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 msgr2=0x55cb649fbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83dad9d640 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 0x55cb649fbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83dad9d640 1 -- 172.21.15.12:0/1936221469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f8e10 msgr2=0x55cb649cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83dad9d640 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f8e10 0x55cb649cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83dad9d640 1 -- 172.21.15.12:0/1936221469 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb646492d0 con 0x55cb649cc2f0 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83dad9d640 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649cc2f0 0x55cb649fc0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f83d4084760 tx=0x7f83d403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b188c794536d4c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83bbfff640 1 -- 172.21.15.12:0/1936221469 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4084c90 con 0x55cb649cc2f0 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83bbfff640 1 -- 172.21.15.12:0/1936221469 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d407a3b0 con 0x55cb649cc2f0 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83bbfff640 1 -- 172.21.15.12:0/1936221469 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d402cb10 con 0x55cb649cc2f0 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83e3a19740 1 -- 172.21.15.12:0/1936221469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649cc2f0 msgr2=0x55cb649fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.911+0000 7f83e3a19740 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649cc2f0 0x55cb649fc0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f83d4084760 tx=0x7f83d403c000 comp rx=0 tx=0).stop 2024-04-15T15:21:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 -- 172.21.15.12:0/1936221469 shutdown_connections 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649cc2f0 0x55cb649fc0e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 0x55cb649fbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 --2- 172.21.15.12:0/1936221469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f8e10 0x55cb649cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 -- 172.21.15.12:0/1936221469 >> 172.21.15.12:0/1936221469 conn(0x55cb649ce020 msgr2=0x55cb64a16ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 -- 172.21.15.12:0/1936221469 shutdown_connections 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 -- 172.21.15.12:0/1936221469 wait complete. 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 -- start start 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.912+0000 7f83e3a19740 1 Processor -- start 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb649ff610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649fb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649e5780 0x55cb649e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83e3a19740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55cb649369c0 con 0x55cb649f8e10 2024-04-15T15:21:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83e3a19740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55cb648f0a10 con 0x55cb649f91f0 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83e3a19740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55cb649d05c0 con 0x55cb649e5780 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83e3a19740 10 client.0 _ll_register_callbacks cb 0x55cb6492c250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83da59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb649ff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83da59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb649ff610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41062/0 (socket says 172.21.15.12:41062) 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83da59c640 1 -- 172.21.15.12:0/2939115528 learned_addr learned my addr 172.21.15.12:0/2939115528 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649fb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83dad9d640 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649e5780 0x55cb649e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83da59c640 1 -- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649e5780 msgr2=0x55cb649e5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83da59c640 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649e5780 0x55cb649e5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83da59c640 1 -- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 msgr2=0x55cb649fb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83da59c640 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649fb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.913+0000 7f83da59c640 1 -- 172.21.15.12:0/2939115528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb647136b0 con 0x55cb649f8e10 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649fb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83da59c640 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb649ff610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0038d0 tx=0x7f83cc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28cae78f9f6147f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83b97fa640 1 -- 172.21.15.12:0/2939115528 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc027060 con 0x55cb649f8e10 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83b97fa640 1 -- 172.21.15.12:0/2939115528 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f83cc010040 con 0x55cb649f8e10 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 1 -- 172.21.15.12:0/2939115528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d407a3b0 con 0x55cb649f8e10 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83b97fa640 1 -- 172.21.15.12:0/2939115528 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f83cc003e50 con 0x55cb649f8e10 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 10 client.4416 fetch_fsmap learned FSMap version 7 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 1 -- 172.21.15.12:0/2939115528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55cb649fcde0 con 0x55cb649f8e10 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83b97fa640 1 -- 172.21.15.12:0/2939115528 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d407a3b0 con 0x55cb649f8e10 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83b97fa640 1 -- 172.21.15.12:0/2939115528 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f83cc00ecc0 con 0x55cb649f8e10 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 10 client.4416 fetch_fsmap finished waiting for FSMap version 7 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 10 client.4416 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:21:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 20 client.4416 populate_metadata read hostname 'smithi012' 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55cb649fa220 0x55cb649fde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 4 client.4416 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.914+0000 7f83e3a19740 1 -- 172.21.15.12:0/2939115528 --> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb649443e0 con 0x55cb649fa220 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.915+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55cb649fa220 0x55cb649fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.915+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55cb649fa220 0x55cb649fde70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cecb5dc1d7ddd91 server_cookie=5ef8b787b9d10d54 in_seq=0 out_seq=0 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.915+0000 7f83b97fa640 10 client.4416 ms_handle_connect on v2:172.21.15.55:6836/1379664958 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83b97fa640 1 -- 172.21.15.12:0/2939115528 <== mds.0 v2:172.21.15.55:6836/1379664958 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb649443e0 con 0x55cb649fa220 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83b97fa640 10 client.4416 handle_command_reply: tid=0 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 client.4416 shutdown 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 -- 172.21.15.12:0/2939115528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 msgr2=0x55cb649ff610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb649ff610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0038d0 tx=0x7f83cc000ed0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 -- 172.21.15.12:0/2939115528 shutdown_connections 2024-04-15T15:21:18.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55cb649fa220 0x55cb649fde70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649e5780 0x55cb649e5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649fb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83b97fa640 0 client.4416 ms_handle_reset on v2:172.21.15.55:6836/1379664958 2024-04-15T15:21:18.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2939115528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb649ff610 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 -- 172.21.15.12:0/2939115528 >> 172.21.15.12:0/2939115528 conn(0x55cb649cc2f0 msgr2=0x55cb649cfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 -- 172.21.15.12:0/2939115528 shutdown_connections 2024-04-15T15:21:18.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 1 -- 172.21.15.12:0/2939115528 wait complete. 2024-04-15T15:21:18.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.916+0000 7f83e3a19740 20 client.4416 trim_cache size 0 max 16384 2024-04-15T15:21:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83e3a19740 1 Processor -- start 2024-04-15T15:21:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83e3a19740 1 -- start start 2024-04-15T15:21:18.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649e5780 0x55cb649fb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb64a296b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 0x55cb649cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83e3a19740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc202260 con 0x55cb649f8e10 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83e3a19740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc2012c0 con 0x55cb649e5780 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83e3a19740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f83dc200620 con 0x55cb649f91f0 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.935+0000 7f83d9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb64a296b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83d9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb64a296b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41064/0 (socket says 172.21.15.12:41064) 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83d9d9b640 1 -- 172.21.15.12:0/3346635148 learned_addr learned my addr 172.21.15.12:0/3346635148 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83da59c640 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649e5780 0x55cb649fb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83d9d9b640 1 -- 172.21.15.12:0/3346635148 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 msgr2=0x55cb649cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83dad9d640 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 0x55cb649cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 0x55cb649cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83d9d9b640 1 -- 172.21.15.12:0/3346635148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649e5780 msgr2=0x55cb649fb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649e5780 0x55cb649fb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83d9d9b640 1 -- 172.21.15.12:0/3346635148 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb649e5f70 con 0x55cb649f8e10 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb64a296b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f83d0003630 tx=0x7f83d0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcde2bdc08adb4a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83bb7fe640 1 -- 172.21.15.12:0/3346635148 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d001c250 con 0x55cb649f8e10 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83bb7fe640 1 -- 172.21.15.12:0/3346635148 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d001c3f0 con 0x55cb649f8e10 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.936+0000 7f83bb7fe640 1 -- 172.21.15.12:0/3346635148 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d001c5e0 con 0x55cb649f8e10 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 -- 172.21.15.12:0/3346635148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 msgr2=0x55cb64a296b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb64a296b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f83d0003630 tx=0x7f83d0008040 comp rx=0 tx=0).stop 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 -- 172.21.15.12:0/3346635148 shutdown_connections 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f91f0 0x55cb649cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649f8e10 0x55cb64a296b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 --2- 172.21.15.12:0/3346635148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649e5780 0x55cb649fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 -- 172.21.15.12:0/3346635148 >> 172.21.15.12:0/3346635148 conn(0x55cb649cc2f0 msgr2=0x55cb649fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 -- 172.21.15.12:0/3346635148 shutdown_connections 2024-04-15T15:21:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 -- 172.21.15.12:0/3346635148 wait complete. 2024-04-15T15:21:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 -- start start 2024-04-15T15:21:18.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.937+0000 7f83e3a19740 1 Processor -- start 2024-04-15T15:21:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649e5780 0x55cb64a0ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f8e10 0x55cb649e8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83e3a19740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83e3a19740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55cb649fa0a0 con 0x55cb649e5780 2024-04-15T15:21:18.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83e3a19740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc200c70 con 0x55cb649f91f0 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83e3a19740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55cb649fa6f0 con 0x55cb649f8e10 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83e3a19740 10 client.0 _ll_register_callbacks cb 0x7f83cc0050d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83da59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649e5780 0x55cb64a0ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83da59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649e5780 0x55cb64a0ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41078/0 (socket says 172.21.15.12:41078) 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83da59c640 1 -- 172.21.15.12:0/2162401666 learned_addr learned my addr 172.21.15.12:0/2162401666 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83da59c640 1 -- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f8e10 msgr2=0x55cb649e8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.938+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f8e10 0x55cb649e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83da59c640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f8e10 0x55cb649e8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83dad9d640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83da59c640 1 -- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 msgr2=0x55cb649e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83da59c640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649e8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83da59c640 1 -- 172.21.15.12:0/2162401666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb649e4ff0 con 0x55cb649e5780 2024-04-15T15:21:18.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f8e10 0x55cb649e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83dad9d640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83da59c640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649e5780 0x55cb64a0ef50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83cc007f00 tx=0x7f83cc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6911f3ce8ef4d936 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83b97fa640 1 -- 172.21.15.12:0/2162401666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc003e50 con 0x55cb649e5780 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83b97fa640 1 -- 172.21.15.12:0/2162401666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f83cc010040 con 0x55cb649e5780 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 1 -- 172.21.15.12:0/2162401666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d001c3f0 con 0x55cb649e5780 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83b97fa640 1 -- 172.21.15.12:0/2162401666 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f83cc03c050 con 0x55cb649e5780 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 10 client.4422 fetch_fsmap learned FSMap version 7 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 1 -- 172.21.15.12:0/2162401666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55cb649d1af0 con 0x55cb649e5780 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83b97fa640 1 -- 172.21.15.12:0/2162401666 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d001c3f0 con 0x55cb649e5780 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83b97fa640 1 -- 172.21.15.12:0/2162401666 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f83cc03f020 con 0x55cb649e5780 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 10 client.4422 fetch_fsmap finished waiting for FSMap version 7 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 10 client.4422 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:21:18.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 20 client.4422 populate_metadata read hostname 'smithi012' 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55cb649fa220 0x55cb649d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 4 client.4422 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.939+0000 7f83e3a19740 1 -- 172.21.15.12:0/2162401666 --> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] -- command(tid 0: {"prefix": "status"}) -- 0x55cb6493e250 con 0x55cb649fa220 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55cb649fa220 0x55cb649d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83d9d9b640 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55cb649fa220 0x55cb649d06a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a288ee818da3e9d8 server_cookie=978d1d7304dec33d in_seq=0 out_seq=0 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83b97fa640 10 client.4422 ms_handle_connect on v2:172.21.15.55:6836/1379664958 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83b97fa640 1 -- 172.21.15.12:0/2162401666 <== mds.0 v2:172.21.15.55:6836/1379664958 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55cb6493e250 con 0x55cb649fa220 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83b97fa640 10 client.4422 handle_command_reply: tid=0 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 client.4422 shutdown 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 -- 172.21.15.12:0/2162401666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649e5780 msgr2=0x55cb64a0ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649e5780 0x55cb64a0ef50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83cc007f00 tx=0x7f83cc013000 comp rx=0 tx=0).stop 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 -- 172.21.15.12:0/2162401666 shutdown_connections 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55cb649fa220 0x55cb649d06a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55cb649f91f0 0x55cb649e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55cb649f8e10 0x55cb649e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83b97fa640 0 client.4422 ms_handle_reset on v2:172.21.15.55:6836/1379664958 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 --2- 172.21.15.12:0/2162401666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55cb649e5780 0x55cb64a0ef50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 -- 172.21.15.12:0/2162401666 >> 172.21.15.12:0/2162401666 conn(0x55cb649cc2f0 msgr2=0x55cb64a0e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 -- 172.21.15.12:0/2162401666 shutdown_connections 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.940+0000 7f83e3a19740 1 -- 172.21.15.12:0/2162401666 wait complete. 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e3a19740 20 client.4422 trim_cache size 0 max 16384 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "928fa63b-2322-46eb-a6b8-ef2b880cd7c9", 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-04-15T15:21:18.949 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4290, 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 7.3140867140000001, 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 7, 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 25, 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 25, 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 14.767501995 2024-04-15T15:21:18.950 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:21:18.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 -- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f83bc06c8f0 msgr2=0x7f83bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f83bc06c8f0 0x7f83bc06edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f83cc001640 tx=0x7f83cc009040 comp rx=0 tx=0).stop 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 -- 172.21.15.12:0/1178793772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc1609e0 msgr2=0x7f83dc0b69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc1609e0 0x7f83dc0b69d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009f70 tx=0x7f83d4004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 -- 172.21.15.12:0/1178793772 shutdown_connections 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83dc1609e0 0x7f83dc0b69d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83dc09b780 0x7f83dc1f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f83bc06c8f0 0x7f83bc06edb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 --2- 172.21.15.12:0/1178793772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83dc09a9d0 0x7f83dc1f80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 -- 172.21.15.12:0/1178793772 >> 172.21.15.12:0/1178793772 conn(0x7f83dc0b3fb0 msgr2=0x7f83dc15de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 -- 172.21.15.12:0/1178793772 shutdown_connections 2024-04-15T15:21:18.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:18.941+0000 7f83e0e37640 1 -- 172.21.15.12:0/1178793772 wait complete. 2024-04-15T15:21:18.956 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:21:18.956 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:21:18.956 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:21:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.058+0000 7f72138c7640 1 Processor -- start 2024-04-15T15:21:19.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.058+0000 7f72138c7640 1 -- start start 2024-04-15T15:21:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.059+0000 7f72138c7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f720c06cd00 0x7f720c0d0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.059+0000 7f72138c7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c15ec70 con 0x7f720c09a9d0 2024-04-15T15:21:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.059+0000 7f72138c7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c0b1580 con 0x7f720c09b720 2024-04-15T15:21:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.059+0000 7f72138c7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f720c15f840 con 0x7f720c06cd00 2024-04-15T15:21:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.059+0000 7f7211663640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f720c09a9d0 0x7f720c09adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58378/0 (socket says 172.21.15.12:58378) 2024-04-15T15:21:19.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.059+0000 7f7211663640 1 -- 172.21.15.12:0/510357347 learned_addr learned my addr 172.21.15.12:0/510357347 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:19.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.059+0000 7f7211e64640 1 --2- 172.21.15.12:0/510357347 >> v2:172.21.15.12:3300/0 conn(0x7f720c06cd00 0x7f720c0d0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:19.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.060+0000 7f720bfff640 1 -- 172.21.15.12:0/510357347 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430747371 0 0) 0x7f720c15ec70 con 0x7f720c09a9d0 2024-04-15T15:21:19.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.060+0000 7f720bfff640 1 -- 172.21.15.12:0/510357347 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71f4003520 con 0x7f720c09a9d0 2024-04-15T15:21:19.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.060+0000 7f7211e64640 1 -- 172.21.15.12:0/510357347 >> v1:172.21.15.55:6789/0 conn(0x7f720c09b720 legacy=0x7f720c0d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:19.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.060+0000 7f7211e64640 1 -- 172.21.15.12:0/510357347 >> v1:172.21.15.12:6789/0 conn(0x7f720c09a9d0 legacy=0x7f720c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:19.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.060+0000 7f7211e64640 1 -- 172.21.15.12:0/510357347 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720c0b12a0 con 0x7f720c06cd00 2024-04-15T15:21:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.060+0000 7f7211e64640 1 --2- 172.21.15.12:0/510357347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c06cd00 0x7f720c0d0e00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f72040097a0 tx=0x7f720402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1ec475bf6e3f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f720bfff640 1 -- 172.21.15.12:0/510357347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7204034070 con 0x7f720c06cd00 2024-04-15T15:21:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f720bfff640 1 -- 172.21.15.12:0/510357347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7204030610 con 0x7f720c06cd00 2024-04-15T15:21:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f720bfff640 1 -- 172.21.15.12:0/510357347 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72040309e0 con 0x7f720c06cd00 2024-04-15T15:21:19.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f72138c7640 1 -- 172.21.15.12:0/510357347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c06cd00 msgr2=0x7f720c0d0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f72138c7640 1 --2- 172.21.15.12:0/510357347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c06cd00 0x7f720c0d0e00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f72040097a0 tx=0x7f720402b960 comp rx=0 tx=0).stop 2024-04-15T15:21:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f72138c7640 1 -- 172.21.15.12:0/510357347 shutdown_connections 2024-04-15T15:21:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f72138c7640 1 --2- 172.21.15.12:0/510357347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c06cd00 0x7f720c0d0e00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f72138c7640 1 -- 172.21.15.12:0/510357347 >> 172.21.15.12:0/510357347 conn(0x7f720c0b3e70 msgr2=0x7f720c0b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f72138c7640 1 -- 172.21.15.12:0/510357347 shutdown_connections 2024-04-15T15:21:19.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.061+0000 7f72138c7640 1 -- 172.21.15.12:0/510357347 wait complete. 2024-04-15T15:21:19.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.062+0000 7f72138c7640 1 Processor -- start 2024-04-15T15:21:19.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.062+0000 7f72138c7640 1 -- start start 2024-04-15T15:21:19.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f72138c7640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f720c06cd00 0x7f720c0c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:19.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f72138c7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c09a9d0 0x7f720c160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:19.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f72138c7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f720c09b720 0x7f720c1614d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:19.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f72138c7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f720c15f4d0 con 0x7f720c09a9d0 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f72138c7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f720c15f1f0 con 0x7f720c09b720 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f72138c7640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f720c15fcd0 con 0x7f720c06cd00 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f7210e62640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c09a9d0 0x7f720c160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f7210e62640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c09a9d0 0x7f720c160f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41106/0 (socket says 172.21.15.12:41106) 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f7210e62640 1 -- 172.21.15.12:0/3895727487 learned_addr learned my addr 172.21.15.12:0/3895727487 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f7211e64640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f720c09b720 0x7f720c1614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.063+0000 7f7211663640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f720c06cd00 0x7f720c0c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7210e62640 1 -- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f720c06cd00 msgr2=0x7f720c0c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7210e62640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f720c06cd00 0x7f720c0c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7210e62640 1 -- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f720c09b720 msgr2=0x7f720c1614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7210e62640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f720c09b720 0x7f720c1614d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7210e62640 1 -- 172.21.15.12:0/3895727487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720c0ca2e0 con 0x7f720c09a9d0 2024-04-15T15:21:19.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7211663640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f720c06cd00 0x7f720c0c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:19.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7210e62640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c09a9d0 0x7f720c160f90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f720000c400 tx=0x7f720000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:19.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7209ffb640 1 -- 172.21.15.12:0/3895727487 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72000052d0 con 0x7f720c09a9d0 2024-04-15T15:21:19.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7209ffb640 1 -- 172.21.15.12:0/3895727487 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7200005490 con 0x7f720c09a9d0 2024-04-15T15:21:19.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f720c161c50 con 0x7f720c09a9d0 2024-04-15T15:21:19.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f7209ffb640 1 -- 172.21.15.12:0/3895727487 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200007260 con 0x7f720c09a9d0 2024-04-15T15:21:19.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.064+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f720c162210 con 0x7f720c09a9d0 2024-04-15T15:21:19.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.065+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f720c0690f0 con 0x7f720c09a9d0 2024-04-15T15:21:19.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.066+0000 7f7209ffb640 1 -- 172.21.15.12:0/3895727487 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7200007400 con 0x7f720c09a9d0 2024-04-15T15:21:19.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.066+0000 7f7209ffb640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f71f806c8f0 0x7f71f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:19.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.066+0000 7f7209ffb640 1 -- 172.21.15.12:0/3895727487 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f720008b370 con 0x7f720c09a9d0 2024-04-15T15:21:19.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.066+0000 7f7211663640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f71f806c8f0 0x7f71f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:19.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.067+0000 7f7211663640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f71f806c8f0 0x7f71f806edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f71fc001560 tx=0x7f71fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:19.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.071+0000 7f7209ffb640 1 -- 172.21.15.12:0/3895727487 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7200058e10 con 0x7f720c09a9d0 2024-04-15T15:21:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.294+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f720c072610 con 0x7f720c09a9d0 2024-04-15T15:21:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.294+0000 7f7209ffb640 1 -- 172.21.15.12:0/3895727487 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f720005d110 con 0x7f720c09a9d0 2024-04-15T15:21:19.296 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:21:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f71f806c8f0 msgr2=0x7f71f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f71f806c8f0 0x7f71f806edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f71fc001560 tx=0x7f71fc009040 comp rx=0 tx=0).stop 2024-04-15T15:21:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c09a9d0 msgr2=0x7f720c160f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c09a9d0 0x7f720c160f90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f720000c400 tx=0x7f720000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:21:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 shutdown_connections 2024-04-15T15:21:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f71f806c8f0 0x7f71f806edb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f720c09b720 0x7f720c1614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:19.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f720c09a9d0 0x7f720c160f90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:19.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 --2- 172.21.15.12:0/3895727487 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f720c06cd00 0x7f720c0c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:19.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 >> 172.21.15.12:0/3895727487 conn(0x7f720c0b3e70 msgr2=0x7f720c0d06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:19.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 shutdown_connections 2024-04-15T15:21:19.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19.296+0000 7f72138c7640 1 -- 172.21.15.12:0/3895727487 wait complete. 2024-04-15T15:21:19.308 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-15T15:21:19.308 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:21:19.308 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-04-15T15:21:19.308 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-15T15:21:19.308 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:21:19.308 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-15T15:21:19.308 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:21:19.309 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:21:19.309 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-15T15:21:19.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:19.309 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-04-15T15:21:19.324 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:19.324 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:19.325 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-04-15T15:21:19.326 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:21:19.326 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42669sec preferred_lft 42669sec 2024-04-15T15:21:19.326 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-04-15T15:21:19.326 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:19.326 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:19.326 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:19.326 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-04-15T15:21:19.326 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:21:19.327 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:19.327 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:19.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:21:19.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-04-15T15:21:19.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-04-15T15:21:19.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:21:19.327 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:19.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:19.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:19.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:19.522 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:21:19.550 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-04-15T15:21:19.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:19.554 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-04-15T15:21:19.625 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-04-15T15:21:19.625 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:21:19.625 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:21:19.625 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:21:19.625 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:21:19.626 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:19.626 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:19.626 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:21:19.626 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:21:19.626 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:21:19.626 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:19.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:19.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:19.954 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-15T15:21:19.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:19.970 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-04-15T15:21:20.026 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:20.026 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:20.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:21:20.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-15T15:21:20.026 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:20.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:20.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:20.150 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-15T15:21:20.151 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:20.151 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:20.151 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-15T15:21:20.151 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:21:20.151 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-15T15:21:20.151 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-15T15:21:20.151 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-15T15:21:20.151 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:20.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:20.294 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:20.294 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:20.294 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-04-15T15:21:20.294 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:21:20.294 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:20.377 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-15T15:21:20.378 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:20.378 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:21:20.392 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-15T15:21:20.393 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:20.393 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:21:20.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:21:20.516 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:21:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:21:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:21:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:21:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:21:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:21:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:21:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:20.573 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:21:20.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:20.575 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:21:20.627 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:21:20.628 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-15T15:21:20.670 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:21:20.698 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:20.755 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:21:20.755 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:21:20.755 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:21:20.755 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:21:20.755 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:21:20.756 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:21:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:21:20.758 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:21:20.758 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:21:20.758 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:21:20.758 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:21:20.758 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:21:20.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:20.758 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:21:20.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:2024-04-15T15:21:20.760+0000 7ff162e45480 -1 init, newargv = 0x5566bb3933c0 newargc=15 2024-04-15T15:21:20.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[44481]: starting ceph client 2024-04-15T15:21:20.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[44481]: starting fuse 2024-04-15T15:21:20.777 INFO:teuthology.orchestra.run.smithi012.stdout:43 2024-04-15T15:21:20.777 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:21:20.777 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:21:20.777 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-04-15T15:21:20.778 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.admin")) 2024-04-15T15:21:20.779 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-04-15T15:21:20.900 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.admin.44481.asok 2024-04-15T15:21:20.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:20.910 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44481.asok 2024-04-15T15:21:20.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:20.910 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44481.asok status 2024-04-15T15:21:21.050 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:21:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-04-15T15:21:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:21:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:21:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "admin", 2024-04-15T15:21:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-04-15T15:21:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "44481", 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4437, 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4437 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:21:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3345687579 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3345687579 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.4437 192.168.0.1:0/3345687579", 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.1:0/3345687579", 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 7, 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 25, 2024-04-15T15:21:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:21:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-04-15T15:21:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-04-15T15:21:21.053 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:21:21.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:21.058 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-04-15T15:21:21.059 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:21:21.074 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-04-15T15:21:21.074 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:21:21.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:21.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:21:21.143 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-15T15:21:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T15:21:21.254 INFO:teuthology.orchestra.run.smithi012.stdout:# file: . 2024-04-15T15:21:21.254 INFO:teuthology.orchestra.run.smithi012.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-15T15:21:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:21.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:21.260 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:21:21.260 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:21:21.276 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:21.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.419+0000 7f4be3759640 1 Processor -- start 2024-04-15T15:21:21.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.419+0000 7f4be3759640 1 -- start start 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be3759640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4bdc0d15e0 0x7f4bdc0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be3759640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bdc15eb70 con 0x7f4bdc15e790 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be3759640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bdc0d3c80 con 0x7f4bdc0d4a40 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be3759640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4bdc0d3790 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be0cf4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4bdc0d15e0 0x7f4bdc0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be14f5640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4bdc15e790 0x7f4bdc0b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58384/0 (socket says 172.21.15.12:58384) 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be14f5640 1 -- 172.21.15.12:0/1999313442 learned_addr learned my addr 172.21.15.12:0/1999313442 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be0cf4640 1 -- 172.21.15.12:0/1999313442 >> v1:172.21.15.55:6789/0 conn(0x7f4bdc0d4a40 legacy=0x7f4bdc0d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be0cf4640 1 -- 172.21.15.12:0/1999313442 >> v1:172.21.15.12:6789/0 conn(0x7f4bdc15e790 legacy=0x7f4bdc0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4bd3fff640 1 -- 172.21.15.12:0/1999313442 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240974243 0 0) 0x7f4bdc15eb70 con 0x7f4bdc15e790 2024-04-15T15:21:21.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.420+0000 7f4be0cf4640 1 -- 172.21.15.12:0/1999313442 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bdc0d3f60 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.421+0000 7f4be0cf4640 1 --2- 172.21.15.12:0/1999313442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc0d1a90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0097a0 tx=0x7f4bcc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2863a970ac8438bf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.421+0000 7f4bd3fff640 1 -- 172.21.15.12:0/1999313442 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0315a0 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.421+0000 7f4bd3fff640 1 -- 172.21.15.12:0/1999313442 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc037070 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.421+0000 7f4bd3fff640 1 -- 172.21.15.12:0/1999313442 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc031960 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.422+0000 7f4be3759640 1 -- 172.21.15.12:0/1999313442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 msgr2=0x7f4bdc0d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.422+0000 7f4be3759640 1 --2- 172.21.15.12:0/1999313442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc0d1a90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0097a0 tx=0x7f4bcc02b840 comp rx=0 tx=0).stop 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.422+0000 7f4be3759640 1 -- 172.21.15.12:0/1999313442 shutdown_connections 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.422+0000 7f4be3759640 1 --2- 172.21.15.12:0/1999313442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc0d1a90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.422+0000 7f4be3759640 1 -- 172.21.15.12:0/1999313442 >> 172.21.15.12:0/1999313442 conn(0x7f4bdc067b40 msgr2=0x7f4bdc067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:21.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.422+0000 7f4be3759640 1 -- 172.21.15.12:0/1999313442 shutdown_connections 2024-04-15T15:21:21.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.422+0000 7f4be3759640 1 -- 172.21.15.12:0/1999313442 wait complete. 2024-04-15T15:21:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.423+0000 7f4be3759640 1 Processor -- start 2024-04-15T15:21:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.423+0000 7f4be3759640 1 -- start start 2024-04-15T15:21:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be3759640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be3759640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bdc0d4a40 0x7f4bdc0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be3759640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4bdc15e790 0x7f4bdc160f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be3759640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4bdc0b7db0 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be3759640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4bdc0d45b0 con 0x7f4bdc0d4a40 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be3759640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f4bdc0d30d0 con 0x7f4bdc15e790 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41130/0 (socket says 172.21.15.12:41130) 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 -- 172.21.15.12:0/524962937 learned_addr learned my addr 172.21.15.12:0/524962937 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be0cf4640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bdc0d4a40 0x7f4bdc0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be1cf6640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4bdc15e790 0x7f4bdc160f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 -- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4bdc15e790 msgr2=0x7f4bdc160f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4bdc15e790 0x7f4bdc160f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 -- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bdc0d4a40 msgr2=0x7f4bdc0c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bdc0d4a40 0x7f4bdc0c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 -- 172.21.15.12:0/524962937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bdc150fb0 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be1cf6640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4bdc15e790 0x7f4bdc160f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.424+0000 7f4be14f5640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc150a70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800c3c0 tx=0x7f4bd800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:21.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.425+0000 7f4bd1ffb640 1 -- 172.21.15.12:0/524962937 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8017070 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.425+0000 7f4bd1ffb640 1 -- 172.21.15.12:0/524962937 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd8005ce0 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.425+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bdc0c93f0 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.425+0000 7f4bd1ffb640 1 -- 172.21.15.12:0/524962937 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8007310 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.425+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bdc0c99b0 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.426+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bdc0689e0 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.426+0000 7f4bd1ffb640 1 -- 172.21.15.12:0/524962937 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4bd801a070 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.427+0000 7f4bd1ffb640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4bc006c8f0 0x7f4bc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.427+0000 7f4bd1ffb640 1 -- 172.21.15.12:0/524962937 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4bd808b2a0 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.427+0000 7f4be0cf4640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4bc006c8f0 0x7f4bc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.427+0000 7f4be0cf4640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4bc006c8f0 0x7f4bc006edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc036000 tx=0x7f4bcc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.431+0000 7f4bd1ffb640 1 -- 172.21.15.12:0/524962937 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4bd8058c10 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.615+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4bdc0b8180 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.617+0000 7f4bd1ffb640 1 -- 172.21.15.12:0/524962937 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f4bd805cf10 con 0x7f4bdc0d15e0 2024-04-15T15:21:21.618 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:21.618 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4bc006c8f0 msgr2=0x7f4bc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4bc006c8f0 0x7f4bc006edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc036000 tx=0x7f4bcc008030 comp rx=0 tx=0).stop 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 msgr2=0x7f4bdc150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc150a70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800c3c0 tx=0x7f4bd800c880 comp rx=0 tx=0).stop 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 shutdown_connections 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4bdc15e790 0x7f4bdc160f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4bc006c8f0 0x7f4bc006edb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bdc0d4a40 0x7f4bdc0c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 --2- 172.21.15.12:0/524962937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bdc0d15e0 0x7f4bdc150a70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 >> 172.21.15.12:0/524962937 conn(0x7f4bdc067b40 msgr2=0x7f4bdc074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 shutdown_connections 2024-04-15T15:21:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.619+0000 7f4be3759640 1 -- 172.21.15.12:0/524962937 wait complete. 2024-04-15T15:21:21.631 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:21.632 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:21.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.738+0000 7fed90538640 1 Processor -- start 2024-04-15T15:21:21.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.738+0000 7fed90538640 1 -- start start 2024-04-15T15:21:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.739+0000 7fed90538640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fed8809b020 0x7fed880ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.739+0000 7fed90538640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed880d14b0 con 0x7fed8809ac40 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.739+0000 7fed90538640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed880cd550 con 0x7fed880b49d0 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.739+0000 7fed90538640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fed880b1b70 con 0x7fed8809b020 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.739+0000 7fed8e2d4640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fed8809ac40 0x7fed880d0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58400/0 (socket says 172.21.15.12:58400) 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.739+0000 7fed8e2d4640 1 -- 172.21.15.12:0/2284343214 learned_addr learned my addr 172.21.15.12:0/2284343214 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8ead5640 1 --2- 172.21.15.12:0/2284343214 >> v2:172.21.15.55:3300/0 conn(0x7fed8809b020 0x7fed880ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4101123702 0 0) 0x7fed880d14b0 con 0x7fed8809ac40 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed70003520 con 0x7fed8809ac40 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 30887549 0 0) 0x7fed70003520 con 0x7fed8809ac40 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 >> v2:172.21.15.55:3300/0 conn(0x7fed8809b020 msgr2=0x7fed880ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 --2- 172.21.15.12:0/2284343214 >> v2:172.21.15.55:3300/0 conn(0x7fed8809b020 0x7fed880ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 >> v1:172.21.15.55:6789/0 conn(0x7fed880b49d0 legacy=0x7fed88160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed880cd830 con 0x7fed8809ac40 2024-04-15T15:21:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fed80001670 con 0x7fed8809ac40 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 >> v1:172.21.15.12:6789/0 conn(0x7fed8809ac40 legacy=0x7fed880d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed700039b0 0x7fed70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed700051c0 0x7fed7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed7000c040 con 0x7fed70004800 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fed7000c1f0 con 0x7fed700039b0 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fed7000c3f0 con 0x7fed700051c0 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8dad3640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8e2d4640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed700039b0 0x7fed70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8ead5640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed700051c0 0x7fed7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8dad3640 1 -- 172.21.15.12:0/2284343214 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed700051c0 msgr2=0x7fed7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8dad3640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed700051c0 0x7fed7000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8dad3640 1 -- 172.21.15.12:0/2284343214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed700039b0 msgr2=0x7fed70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8dad3640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed700039b0 0x7fed70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8dad3640 1 -- 172.21.15.12:0/2284343214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed70003520 con 0x7fed70004800 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.740+0000 7fed8ead5640 1 -- 172.21.15.12:0/2284343214 reap_dead start 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.741+0000 7fed8dad3640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fed7c001750 tx=0x7fed70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85a76d494f8e7cf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.741+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c001370 con 0x7fed70004800 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.741+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed7c0052d0 con 0x7fed70004800 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.741+0000 7fed8d2d2640 1 -- 172.21.15.12:0/2284343214 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c003830 con 0x7fed70004800 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.742+0000 7fed90538640 1 -- 172.21.15.12:0/2284343214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 msgr2=0x7fed70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.742+0000 7fed90538640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fed7c001750 tx=0x7fed70005830 comp rx=0 tx=0).stop 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.742+0000 7fed90538640 1 -- 172.21.15.12:0/2284343214 shutdown_connections 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.742+0000 7fed90538640 1 --2- 172.21.15.12:0/2284343214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed70004800 0x7fed70004c80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.742+0000 7fed90538640 1 -- 172.21.15.12:0/2284343214 >> 172.21.15.12:0/2284343214 conn(0x7fed880b3eb0 msgr2=0x7fed880b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.742+0000 7fed90538640 1 -- 172.21.15.12:0/2284343214 shutdown_connections 2024-04-15T15:21:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.742+0000 7fed90538640 1 -- 172.21.15.12:0/2284343214 wait complete. 2024-04-15T15:21:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.743+0000 7fed90538640 1 Processor -- start 2024-04-15T15:21:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.743+0000 7fed90538640 1 -- start start 2024-04-15T15:21:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed90538640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed70004800 0x7fed8814c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed90538640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed880b4fe0 0x7fed88070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed90538640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed880b5460 0x7fed88071330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed90538640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed880d0e50 con 0x7fed880b4fe0 2024-04-15T15:21:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed90538640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fed880d0cd0 con 0x7fed880b5460 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed90538640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fed880d0fd0 con 0x7fed70004800 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8dad3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed880b4fe0 0x7fed88070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8dad3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed880b4fe0 0x7fed88070df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41140/0 (socket says 172.21.15.12:41140) 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8dad3640 1 -- 172.21.15.12:0/1422610253 learned_addr learned my addr 172.21.15.12:0/1422610253 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8e2d4640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed70004800 0x7fed8814c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8ead5640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed880b5460 0x7fed88071330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8dad3640 1 -- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed70004800 msgr2=0x7fed8814c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8dad3640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed70004800 0x7fed8814c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8dad3640 1 -- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed880b5460 msgr2=0x7fed88071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8dad3640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed880b5460 0x7fed88071330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.744+0000 7fed8dad3640 1 -- 172.21.15.12:0/1422610253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8814ccf0 con 0x7fed880b4fe0 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.745+0000 7fed8dad3640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed880b4fe0 0x7fed88070df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fed7c000c00 tx=0x7fed7c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.745+0000 7fed86ffd640 1 -- 172.21.15.12:0/1422610253 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c041070 con 0x7fed880b4fe0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.745+0000 7fed86ffd640 1 -- 172.21.15.12:0/1422610253 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed7c005480 con 0x7fed880b4fe0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.745+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed880c96e0 con 0x7fed880b4fe0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.745+0000 7fed86ffd640 1 -- 172.21.15.12:0/1422610253 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c001370 con 0x7fed880b4fe0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.745+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed880c9ca0 con 0x7fed880b4fe0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.746+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed880c9450 con 0x7fed880b4fe0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.746+0000 7fed86ffd640 1 -- 172.21.15.12:0/1422610253 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed7c044070 con 0x7fed880b4fe0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.747+0000 7fed86ffd640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fed7406c920 0x7fed7406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.747+0000 7fed86ffd640 1 -- 172.21.15.12:0/1422610253 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fed7c0ad7a0 con 0x7fed880b4fe0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.747+0000 7fed8e2d4640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fed7406c920 0x7fed7406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.747+0000 7fed8e2d4640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fed7406c920 0x7fed7406ede0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fed800011e0 tx=0x7fed80000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.751+0000 7fed86ffd640 1 -- 172.21.15.12:0/1422610253 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed7c07b360 con 0x7fed880b4fe0 2024-04-15T15:21:21.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.935+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fed880689e0 con 0x7fed880b4fe0 2024-04-15T15:21:21.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.935+0000 7fed86ffd640 1 -- 172.21.15.12:0/1422610253 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fed7c07f660 con 0x7fed880b4fe0 2024-04-15T15:21:21.936 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:21.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fed7406c920 msgr2=0x7fed7406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fed7406c920 0x7fed7406ede0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fed800011e0 tx=0x7fed80000ed0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed880b4fe0 msgr2=0x7fed88070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed880b4fe0 0x7fed88070df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fed7c000c00 tx=0x7fed7c001120 comp rx=0 tx=0).stop 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 shutdown_connections 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fed7406c920 0x7fed7406ede0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed880b5460 0x7fed88071330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed880b4fe0 0x7fed88070df0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 --2- 172.21.15.12:0/1422610253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fed70004800 0x7fed8814c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 >> 172.21.15.12:0/1422610253 conn(0x7fed880b3eb0 msgr2=0x7fed880b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 shutdown_connections 2024-04-15T15:21:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:21.937+0000 7fed90538640 1 -- 172.21.15.12:0/1422610253 wait complete. 2024-04-15T15:21:21.949 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.055+0000 7ffb5ff32640 1 Processor -- start 2024-04-15T15:21:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.055+0000 7ffb5ff32640 1 -- start start 2024-04-15T15:21:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5ff32640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ffb5815e790 0x7ffb580b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5ff32640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb5815eb70 con 0x7ffb580b7be0 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5ff32640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb580d3c80 con 0x7ffb580d1cf0 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5ff32640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ffb580d3790 con 0x7ffb5815e790 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5d4cd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ffb580b7be0 0x7ffb580d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58416/0 (socket says 172.21.15.12:58416) 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5d4cd640 1 -- 172.21.15.12:0/1739445605 learned_addr learned my addr 172.21.15.12:0/1739445605 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5dcce640 1 --2- 172.21.15.12:0/1739445605 >> v2:172.21.15.55:3300/0 conn(0x7ffb5815e790 0x7ffb580b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5cccc640 1 -- 172.21.15.12:0/1739445605 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346715818 0 0) 0x7ffb5815eb70 con 0x7ffb580b7be0 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5cccc640 1 -- 172.21.15.12:0/1739445605 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb44003520 con 0x7ffb580b7be0 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5dcce640 1 -- 172.21.15.12:0/1739445605 >> v1:172.21.15.55:6789/0 conn(0x7ffb580d1cf0 legacy=0x7ffb580d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5cccc640 1 -- 172.21.15.12:0/1739445605 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3820929911 0 0) 0x7ffb44003520 con 0x7ffb580b7be0 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5dcce640 1 -- 172.21.15.12:0/1739445605 >> v1:172.21.15.12:6789/0 conn(0x7ffb580b7be0 legacy=0x7ffb580d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.056+0000 7ffb5dcce640 1 -- 172.21.15.12:0/1739445605 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb580d3f60 con 0x7ffb5815e790 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.057+0000 7ffb5dcce640 1 --2- 172.21.15.12:0/1739445605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb5815e790 0x7ffb580b76a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb540097a0 tx=0x7ffb5402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b73fcf809fac7f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.057+0000 7ffb5cccc640 1 -- 172.21.15.12:0/1739445605 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540315a0 con 0x7ffb5815e790 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.057+0000 7ffb5cccc640 1 -- 172.21.15.12:0/1739445605 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb54037070 con 0x7ffb5815e790 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.057+0000 7ffb5cccc640 1 -- 172.21.15.12:0/1739445605 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54031940 con 0x7ffb5815e790 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.058+0000 7ffb5ff32640 1 -- 172.21.15.12:0/1739445605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb5815e790 msgr2=0x7ffb580b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.058+0000 7ffb5ff32640 1 --2- 172.21.15.12:0/1739445605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb5815e790 0x7ffb580b76a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb540097a0 tx=0x7ffb5402b840 comp rx=0 tx=0).stop 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.058+0000 7ffb5ff32640 1 -- 172.21.15.12:0/1739445605 shutdown_connections 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.058+0000 7ffb5ff32640 1 --2- 172.21.15.12:0/1739445605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb5815e790 0x7ffb580b76a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.058+0000 7ffb5ff32640 1 -- 172.21.15.12:0/1739445605 >> 172.21.15.12:0/1739445605 conn(0x7ffb58067b40 msgr2=0x7ffb58067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.058+0000 7ffb5ff32640 1 -- 172.21.15.12:0/1739445605 shutdown_connections 2024-04-15T15:21:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.058+0000 7ffb5ff32640 1 -- 172.21.15.12:0/1739445605 wait complete. 2024-04-15T15:21:22.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.059+0000 7ffb5ff32640 1 Processor -- start 2024-04-15T15:21:22.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.059+0000 7ffb5ff32640 1 -- start start 2024-04-15T15:21:22.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5ff32640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb580b7be0 0x7ffb580cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5ff32640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb580d1cf0 0x7ffb581509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5ff32640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ffb5815e790 0x7ffb580cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5ff32640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffb580d5150 con 0x7ffb580d1cf0 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5ff32640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ffb580d45b0 con 0x7ffb580b7be0 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5ff32640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ffb580d52d0 con 0x7ffb5815e790 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb580d1cf0 0x7ffb581509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb580d1cf0 0x7ffb581509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41148/0 (socket says 172.21.15.12:41148) 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 -- 172.21.15.12:0/3162555518 learned_addr learned my addr 172.21.15.12:0/3162555518 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5dcce640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb580b7be0 0x7ffb580cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5e4cf640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ffb5815e790 0x7ffb580cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 -- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ffb5815e790 msgr2=0x7ffb580cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ffb5815e790 0x7ffb580cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 -- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb580b7be0 msgr2=0x7ffb580cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb580b7be0 0x7ffb580cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 -- 172.21.15.12:0/3162555518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb58150f20 con 0x7ffb580d1cf0 2024-04-15T15:21:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb5d4cd640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb580d1cf0 0x7ffb581509e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb48001eb0 tx=0x7ffb4800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.060+0000 7ffb4e7fc640 1 -- 172.21.15.12:0/3162555518 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb480074e0 con 0x7ffb580d1cf0 2024-04-15T15:21:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.061+0000 7ffb4e7fc640 1 -- 172.21.15.12:0/3162555518 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb480047e0 con 0x7ffb580d1cf0 2024-04-15T15:21:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.061+0000 7ffb4e7fc640 1 -- 172.21.15.12:0/3162555518 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb48004c10 con 0x7ffb580d1cf0 2024-04-15T15:21:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.061+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb580ccdf0 con 0x7ffb580d1cf0 2024-04-15T15:21:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.061+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb580cd3b0 con 0x7ffb580d1cf0 2024-04-15T15:21:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.062+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb580689e0 con 0x7ffb580d1cf0 2024-04-15T15:21:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.062+0000 7ffb4e7fc640 1 -- 172.21.15.12:0/3162555518 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffb48011070 con 0x7ffb580d1cf0 2024-04-15T15:21:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.062+0000 7ffb4e7fc640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ffb3c06c8f0 0x7ffb3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.063+0000 7ffb4e7fc640 1 -- 172.21.15.12:0/3162555518 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ffb4808b910 con 0x7ffb580d1cf0 2024-04-15T15:21:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.063+0000 7ffb5dcce640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ffb3c06c8f0 0x7ffb3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.063+0000 7ffb5dcce640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ffb3c06c8f0 0x7ffb3c06edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb54036000 tx=0x7ffb5403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.067+0000 7ffb4e7fc640 1 -- 172.21.15.12:0/3162555518 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffb48059320 con 0x7ffb580d1cf0 2024-04-15T15:21:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.250+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffb580b8cc0 con 0x7ffb580d1cf0 2024-04-15T15:21:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.250+0000 7ffb4e7fc640 1 -- 172.21.15.12:0/3162555518 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7ffb4805d620 con 0x7ffb580d1cf0 2024-04-15T15:21:22.252 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ffb3c06c8f0 msgr2=0x7ffb3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ffb3c06c8f0 0x7ffb3c06edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb54036000 tx=0x7ffb5403d040 comp rx=0 tx=0).stop 2024-04-15T15:21:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb580d1cf0 msgr2=0x7ffb581509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb580d1cf0 0x7ffb581509e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb48001eb0 tx=0x7ffb4800c870 comp rx=0 tx=0).stop 2024-04-15T15:21:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 shutdown_connections 2024-04-15T15:21:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ffb5815e790 0x7ffb580cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb580d1cf0 0x7ffb581509e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ffb3c06c8f0 0x7ffb3c06edb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 --2- 172.21.15.12:0/3162555518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffb580b7be0 0x7ffb580cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 >> 172.21.15.12:0/3162555518 conn(0x7ffb58067b40 msgr2=0x7ffb580b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:22.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 shutdown_connections 2024-04-15T15:21:22.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.252+0000 7ffb5ff32640 1 -- 172.21.15.12:0/3162555518 wait complete. 2024-04-15T15:21:22.264 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:22.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.370+0000 7f54982c8640 1 Processor -- start 2024-04-15T15:21:22.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.370+0000 7f54982c8640 1 -- start start 2024-04-15T15:21:22.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f54982c8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f549009b880 0x7f5490070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f54982c8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54900d0da0 0x7f54900d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f54982c8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54900d05e0 con 0x7f5490070940 2024-04-15T15:21:22.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f54982c8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f549009bce0 con 0x7f54900d0da0 2024-04-15T15:21:22.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f54982c8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5490071260 con 0x7f549009b880 2024-04-15T15:21:22.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f5496865640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54900d0da0 0x7f54900d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f5496865640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f54900d0da0 0x7f54900d11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52812/0 (socket says 172.21.15.12:52812) 2024-04-15T15:21:22.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f5496865640 1 -- 172.21.15.12:0/1826410893 learned_addr learned my addr 172.21.15.12:0/1826410893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:22.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.371+0000 7f5495863640 1 --2- 172.21.15.12:0/1826410893 >> v2:172.21.15.55:3300/0 conn(0x7f549009b880 0x7f5490070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.372+0000 7f5496865640 1 -- 172.21.15.12:0/1826410893 >> v2:172.21.15.55:3300/0 conn(0x7f549009b880 msgr2=0x7f5490070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:22.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.372+0000 7f5496865640 1 --2- 172.21.15.12:0/1826410893 >> v2:172.21.15.55:3300/0 conn(0x7f549009b880 0x7f5490070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.372+0000 7f5496865640 1 -- 172.21.15.12:0/1826410893 >> v1:172.21.15.55:6789/0 conn(0x7f5490070940 legacy=0x7f5490160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.372+0000 7f5496865640 1 -- 172.21.15.12:0/1826410893 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549015f720 con 0x7f54900d0da0 2024-04-15T15:21:22.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.372+0000 7f5496865640 1 --2- 172.21.15.12:0/1826410893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54900d0da0 0x7f54900d11d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f54900734f0 tx=0x7f548802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fde9c442052a8d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f5495062640 1 -- 172.21.15.12:0/1826410893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548802cb20 con 0x7f54900d0da0 2024-04-15T15:21:22.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f5495062640 1 -- 172.21.15.12:0/1826410893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f548802ccc0 con 0x7f54900d0da0 2024-04-15T15:21:22.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f5495062640 1 -- 172.21.15.12:0/1826410893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5488034610 con 0x7f54900d0da0 2024-04-15T15:21:22.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f54982c8640 1 -- 172.21.15.12:0/1826410893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54900d0da0 msgr2=0x7f54900d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f54982c8640 1 --2- 172.21.15.12:0/1826410893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54900d0da0 0x7f54900d11d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f54900734f0 tx=0x7f548802bf80 comp rx=0 tx=0).stop 2024-04-15T15:21:22.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f54982c8640 1 -- 172.21.15.12:0/1826410893 shutdown_connections 2024-04-15T15:21:22.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f54982c8640 1 --2- 172.21.15.12:0/1826410893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f54900d0da0 0x7f54900d11d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f54982c8640 1 --2- 172.21.15.12:0/1826410893 >> v2:172.21.15.55:3300/0 conn(0x7f549009b880 0x7f5490070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f54982c8640 1 -- 172.21.15.12:0/1826410893 >> 172.21.15.12:0/1826410893 conn(0x7f54900b3f40 msgr2=0x7f54900b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:22.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f54982c8640 1 -- 172.21.15.12:0/1826410893 shutdown_connections 2024-04-15T15:21:22.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.373+0000 7f54982c8640 1 -- 172.21.15.12:0/1826410893 wait complete. 2024-04-15T15:21:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.374+0000 7f54982c8640 1 Processor -- start 2024-04-15T15:21:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.374+0000 7f54982c8640 1 -- start start 2024-04-15T15:21:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f54982c8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5490070940 0x7f549009f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f54982c8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f549009b880 0x7f54900b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f54982c8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54900d0da0 0x7f5490160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f54982c8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f54900d0880 con 0x7f5490070940 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f54982c8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f549015fce0 con 0x7f54900d0da0 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f54982c8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f54900d1710 con 0x7f549009b880 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f5496064640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5490070940 0x7f549009f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f5496064640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5490070940 0x7f549009f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52814/0 (socket says 172.21.15.12:52814) 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f5496064640 1 -- 172.21.15.12:0/2000818088 learned_addr learned my addr 172.21.15.12:0/2000818088 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f5496865640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54900d0da0 0x7f5490160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.375+0000 7f5495863640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f549009b880 0x7f54900b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f5496064640 1 -- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f549009b880 msgr2=0x7f54900b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f5496064640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f549009b880 0x7f54900b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f5496064640 1 -- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54900d0da0 msgr2=0x7f5490160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f5496064640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54900d0da0 0x7f5490160f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f5496064640 1 -- 172.21.15.12:0/2000818088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549009ff20 con 0x7f5490070940 2024-04-15T15:21:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f5496064640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5490070940 0x7f549009f9e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f548000c430 tx=0x7f548000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f548effd640 1 -- 172.21.15.12:0/2000818088 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54800052d0 con 0x7f5490070940 2024-04-15T15:21:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f548effd640 1 -- 172.21.15.12:0/2000818088 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5480005490 con 0x7f5490070940 2024-04-15T15:21:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f548effd640 1 -- 172.21.15.12:0/2000818088 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5480007260 con 0x7f5490070940 2024-04-15T15:21:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54900b7c50 con 0x7f5490070940 2024-04-15T15:21:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.376+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549014f4a0 con 0x7f5490070940 2024-04-15T15:21:22.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.377+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54900689e0 con 0x7f5490070940 2024-04-15T15:21:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.378+0000 7f548effd640 1 -- 172.21.15.12:0/2000818088 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5480007400 con 0x7f5490070940 2024-04-15T15:21:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.378+0000 7f548effd640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f547c06c8f0 0x7f547c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.378+0000 7f548effd640 1 -- 172.21.15.12:0/2000818088 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f548008b2b0 con 0x7f5490070940 2024-04-15T15:21:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.378+0000 7f5495863640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f547c06c8f0 0x7f547c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.379+0000 7f5495863640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f547c06c8f0 0x7f547c06edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5490152520 tx=0x7f5484009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.382+0000 7f548effd640 1 -- 172.21.15.12:0/2000818088 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5480058de0 con 0x7f5490070940 2024-04-15T15:21:22.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.568+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5490070720 con 0x7f5490070940 2024-04-15T15:21:22.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.568+0000 7f548effd640 1 -- 172.21.15.12:0/2000818088 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f548005d0e0 con 0x7f5490070940 2024-04-15T15:21:22.570 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f547c06c8f0 msgr2=0x7f547c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f547c06c8f0 0x7f547c06edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5490152520 tx=0x7f5484009190 comp rx=0 tx=0).stop 2024-04-15T15:21:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5490070940 msgr2=0x7f549009f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5490070940 0x7f549009f9e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f548000c430 tx=0x7f548000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 shutdown_connections 2024-04-15T15:21:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f547c06c8f0 0x7f547c06edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54900d0da0 0x7f5490160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f549009b880 0x7f54900b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 --2- 172.21.15.12:0/2000818088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5490070940 0x7f549009f9e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.570+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 >> 172.21.15.12:0/2000818088 conn(0x7f54900b3f40 msgr2=0x7f54900b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.571+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 shutdown_connections 2024-04-15T15:21:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.571+0000 7f54982c8640 1 -- 172.21.15.12:0/2000818088 wait complete. 2024-04-15T15:21:22.583 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:22.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.689+0000 7fa0d2674640 1 Processor -- start 2024-04-15T15:21:22.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.689+0000 7fa0d2674640 1 -- start start 2024-04-15T15:21:22.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.690+0000 7fa0d2674640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc15e790 0x7fa0cc0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.690+0000 7fa0d2674640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0cc0d1610 0x7fa0cc0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.690+0000 7fa0d2674640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0cc0d2480 con 0x7fa0cc0cfd20 2024-04-15T15:21:22.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.690+0000 7fa0d2674640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa0cc0d2db0 con 0x7fa0cc0d1610 2024-04-15T15:21:22.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.690+0000 7fa0d2674640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa0cc0d2760 con 0x7fa0cc15e790 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.690+0000 7fa0d0c11640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0cc0d1610 0x7fa0cc0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.690+0000 7fa0cbfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc15e790 0x7fa0cc0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0d0c11640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0cc0d1610 0x7fa0cc0d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:43700/0 (socket says 172.21.15.12:43700) 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0cbfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc15e790 0x7fa0cc0d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33578/0 (socket says 172.21.15.12:33578) 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0d0c11640 1 -- 172.21.15.12:0/1210952991 learned_addr learned my addr 172.21.15.12:0/1210952991 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0c3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa0cc0cfd20 0x7fa0cc0d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:34582/0 (socket says 172.21.15.12:34582) 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0d0c11640 1 -- 172.21.15.12:0/1210952991 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc15e790 msgr2=0x7fa0cc0d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0d0c11640 1 --2- 172.21.15.12:0/1210952991 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc15e790 0x7fa0cc0d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0d0c11640 1 -- 172.21.15.12:0/1210952991 >> v1:172.21.15.55:6789/0 conn(0x7fa0cc0cfd20 legacy=0x7fa0cc0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0d0c11640 1 -- 172.21.15.12:0/1210952991 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0cc0d2a40 con 0x7fa0cc0d1610 2024-04-15T15:21:22.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.691+0000 7fa0cbfff640 1 --2- 172.21.15.12:0/1210952991 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc15e790 0x7fa0cc0d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.692+0000 7fa0d0c11640 1 --2- 172.21.15.12:0/1210952991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 0x7fa0cc0d0160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009e50 tx=0x7fa0c402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8db42c4e28c31571 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.692+0000 7fa0cb7fe640 1 -- 172.21.15.12:0/1210952991 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c402cb00 con 0x7fa0cc0d1610 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.692+0000 7fa0cb7fe640 1 -- 172.21.15.12:0/1210952991 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c402cca0 con 0x7fa0cc0d1610 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.692+0000 7fa0cb7fe640 1 -- 172.21.15.12:0/1210952991 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4034610 con 0x7fa0cc0d1610 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.692+0000 7fa0d2674640 1 -- 172.21.15.12:0/1210952991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 msgr2=0x7fa0cc0d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.692+0000 7fa0d2674640 1 --2- 172.21.15.12:0/1210952991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 0x7fa0cc0d0160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009e50 tx=0x7fa0c402bf60 comp rx=0 tx=0).stop 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.692+0000 7fa0d2674640 1 -- 172.21.15.12:0/1210952991 shutdown_connections 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.693+0000 7fa0d2674640 1 --2- 172.21.15.12:0/1210952991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 0x7fa0cc0d0160 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.693+0000 7fa0d2674640 1 --2- 172.21.15.12:0/1210952991 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc15e790 0x7fa0cc0d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.693+0000 7fa0d2674640 1 -- 172.21.15.12:0/1210952991 >> 172.21.15.12:0/1210952991 conn(0x7fa0cc08bf60 msgr2=0x7fa0cc0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.693+0000 7fa0d2674640 1 -- 172.21.15.12:0/1210952991 shutdown_connections 2024-04-15T15:21:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.693+0000 7fa0d2674640 1 -- 172.21.15.12:0/1210952991 wait complete. 2024-04-15T15:21:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.694+0000 7fa0d2674640 1 Processor -- start 2024-04-15T15:21:22.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.694+0000 7fa0d2674640 1 -- start start 2024-04-15T15:21:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.694+0000 7fa0d2674640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc0cfd20 0x7fa0cc0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.694+0000 7fa0d2674640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 0x7fa0cc0cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.694+0000 7fa0d2674640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc15e790 0x7fa0cc157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.694+0000 7fa0d2674640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc15ecf0 con 0x7fa0cc15e790 2024-04-15T15:21:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.694+0000 7fa0d2674640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc15eb70 con 0x7fa0cc0d1610 2024-04-15T15:21:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.694+0000 7fa0d2674640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa0cc0b74f0 con 0x7fa0cc0cfd20 2024-04-15T15:21:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc15e790 0x7fa0cc157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc15e790 0x7fa0cc157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52826/0 (socket says 172.21.15.12:52826) 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 -- 172.21.15.12:0/273044541 learned_addr learned my addr 172.21.15.12:0/273044541 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0c3fff640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 0x7fa0cc0cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0cbfff640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc0cfd20 0x7fa0cc0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 -- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc0cfd20 msgr2=0x7fa0cc0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc0cfd20 0x7fa0cc0c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 -- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 msgr2=0x7fa0cc0cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 0x7fa0cc0cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 -- 172.21.15.12:0/273044541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0cc0c84c0 con 0x7fa0cc15e790 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0cbfff640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc0cfd20 0x7fa0cc0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d0c11640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc15e790 0x7fa0cc157740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4036040 tx=0x7fa0c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0c97fa640 1 -- 172.21.15.12:0/273044541 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4043020 con 0x7fa0cc15e790 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0c97fa640 1 -- 172.21.15.12:0/273044541 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c402ce50 con 0x7fa0cc15e790 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0cc0c87a0 con 0x7fa0cc15e790 2024-04-15T15:21:22.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0c97fa640 1 -- 172.21.15.12:0/273044541 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4009510 con 0x7fa0cc15e790 2024-04-15T15:21:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.695+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0cc0c8d60 con 0x7fa0cc15e790 2024-04-15T15:21:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.696+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0cc0689e0 con 0x7fa0cc15e790 2024-04-15T15:21:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.697+0000 7fa0c97fa640 1 -- 172.21.15.12:0/273044541 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa0c4046020 con 0x7fa0cc15e790 2024-04-15T15:21:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.697+0000 7fa0c97fa640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0ac06c8f0 0x7fa0ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.698+0000 7fa0c97fa640 1 -- 172.21.15.12:0/273044541 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa0c40ae310 con 0x7fa0cc15e790 2024-04-15T15:21:22.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.698+0000 7fa0cbfff640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0ac06c8f0 0x7fa0ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:22.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.698+0000 7fa0cbfff640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0ac06c8f0 0x7fa0ac06edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009af0 tx=0x7fa0bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:22.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.702+0000 7fa0c97fa640 1 -- 172.21.15.12:0/273044541 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0c404d030 con 0x7fa0cc15e790 2024-04-15T15:21:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.887+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa0cc0d1ce0 con 0x7fa0cc15e790 2024-04-15T15:21:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.888+0000 7fa0c97fa640 1 -- 172.21.15.12:0/273044541 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fa0c407c490 con 0x7fa0cc15e790 2024-04-15T15:21:22.889 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.889+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0ac06c8f0 msgr2=0x7fa0ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.889+0000 7fa0d2674640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0ac06c8f0 0x7fa0ac06edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009af0 tx=0x7fa0bc008040 comp rx=0 tx=0).stop 2024-04-15T15:21:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.889+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc15e790 msgr2=0x7fa0cc157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.889+0000 7fa0d2674640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc15e790 0x7fa0cc157740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4036040 tx=0x7fa0c4004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.890+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 shutdown_connections 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.890+0000 7fa0d2674640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0cc15e790 0x7fa0cc157740 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.890+0000 7fa0d2674640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0ac06c8f0 0x7fa0ac06edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.890+0000 7fa0d2674640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0cc0d1610 0x7fa0cc0cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.890+0000 7fa0d2674640 1 --2- 172.21.15.12:0/273044541 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0cc0cfd20 0x7fa0cc0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.890+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 >> 172.21.15.12:0/273044541 conn(0x7fa0cc08bf60 msgr2=0x7fa0cc0d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.890+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 shutdown_connections 2024-04-15T15:21:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:22.890+0000 7fa0d2674640 1 -- 172.21.15.12:0/273044541 wait complete. 2024-04-15T15:21:22.902 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:21:23.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.006+0000 7faa32331640 1 Processor -- start 2024-04-15T15:21:23.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.007+0000 7faa32331640 1 -- start start 2024-04-15T15:21:23.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.007+0000 7faa32331640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7faa2c15e700 0x7faa2c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.007+0000 7faa32331640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa2c0b47a0 con 0x7faa2c0d2360 2024-04-15T15:21:23.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.007+0000 7faa32331640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa2c0b4a80 con 0x7faa2c0d3270 2024-04-15T15:21:23.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.007+0000 7faa32331640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7faa2c0b53b0 con 0x7faa2c15e700 2024-04-15T15:21:23.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.007+0000 7faa23fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7faa2c0d2360 0x7faa2c0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36444/0 (socket says 172.21.15.12:36444) 2024-04-15T15:21:23.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.007+0000 7faa23fff640 1 -- 172.21.15.12:0/3156758592 learned_addr learned my addr 172.21.15.12:0/3156758592 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:23.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2bfff640 1 --2- 172.21.15.12:0/3156758592 >> v2:172.21.15.55:3300/0 conn(0x7faa2c15e700 0x7faa2c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 495995397 0 0) 0x7faa2c0b47a0 con 0x7faa2c0d2360 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa18003520 con 0x7faa2c0d2360 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3009311308 0 0) 0x7faa18003520 con 0x7faa2c0d2360 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 >> v2:172.21.15.55:3300/0 conn(0x7faa2c15e700 msgr2=0x7faa2c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 --2- 172.21.15.12:0/3156758592 >> v2:172.21.15.55:3300/0 conn(0x7faa2c15e700 0x7faa2c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 >> v1:172.21.15.55:6789/0 conn(0x7faa2c0d3270 legacy=0x7faa2c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2c0b4d60 con 0x7faa2c0d2360 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7faa14001670 con 0x7faa2c0d2360 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 >> v1:172.21.15.12:6789/0 conn(0x7faa2c0d2360 legacy=0x7faa2c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa180039b0 0x7faa18003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa18004800 0x7faa18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c0b53b0 con 0x7faa180051c0 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faa1800c090 con 0x7faa18004800 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faa1800c290 con 0x7faa180039b0 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa308ce640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa23fff640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa18004800 0x7faa18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa2bfff640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa180039b0 0x7faa18003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa308ce640 1 -- 172.21.15.12:0/3156758592 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa180039b0 msgr2=0x7faa18003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa308ce640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa180039b0 0x7faa18003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa308ce640 1 -- 172.21.15.12:0/3156758592 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa18004800 msgr2=0x7faa18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:23.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.008+0000 7faa308ce640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa18004800 0x7faa18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.009+0000 7faa308ce640 1 -- 172.21.15.12:0/3156758592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa18003520 con 0x7faa180051c0 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.009+0000 7faa308ce640 1 -- 172.21.15.12:0/3156758592 reap_dead start 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.009+0000 7faa308ce640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa1800b9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faa24001e50 tx=0x7faa18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8de61e13c99204 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.009+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24001a70 con 0x7faa180051c0 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.009+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa24003bb0 con 0x7faa180051c0 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.009+0000 7faa2b7fe640 1 -- 172.21.15.12:0/3156758592 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa240050d0 con 0x7faa180051c0 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.010+0000 7faa32331640 1 -- 172.21.15.12:0/3156758592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 msgr2=0x7faa1800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.010+0000 7faa32331640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa1800b9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faa24001e50 tx=0x7faa18005830 comp rx=0 tx=0).stop 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.010+0000 7faa32331640 1 -- 172.21.15.12:0/3156758592 shutdown_connections 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.010+0000 7faa32331640 1 --2- 172.21.15.12:0/3156758592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa1800b9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.010+0000 7faa32331640 1 -- 172.21.15.12:0/3156758592 >> 172.21.15.12:0/3156758592 conn(0x7faa2c08bf60 msgr2=0x7faa2c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.010+0000 7faa32331640 1 -- 172.21.15.12:0/3156758592 shutdown_connections 2024-04-15T15:21:23.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.010+0000 7faa32331640 1 -- 172.21.15.12:0/3156758592 wait complete. 2024-04-15T15:21:23.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.011+0000 7faa32331640 1 Processor -- start 2024-04-15T15:21:23.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.011+0000 7faa32331640 1 -- start start 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa32331640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa2c14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa32331640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa2c1f7a30 0x7faa2c1f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa32331640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa2c1608e0 0x7faa2c160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa32331640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c0d1d40 con 0x7faa180051c0 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa32331640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c0b4450 con 0x7faa2c1f7a30 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa32331640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faa2c0d1ec0 con 0x7faa2c1608e0 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa2c14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa2c14b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52846/0 (socket says 172.21.15.12:52846) 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 -- 172.21.15.12:0/4247031492 learned_addr learned my addr 172.21.15.12:0/4247031492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:23.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa23fff640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa2c1f7a30 0x7faa2c1f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa308ce640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa2c1608e0 0x7faa2c160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 -- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa2c1608e0 msgr2=0x7faa2c160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa2c1608e0 0x7faa2c160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 -- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa2c1f7a30 msgr2=0x7faa2c1f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa2c1f7a30 0x7faa2c1f7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 -- 172.21.15.12:0/4247031492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2c15ae10 con 0x7faa180051c0 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa23fff640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa2c1f7a30 0x7faa2c1f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.012+0000 7faa2bfff640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa2c14b0a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa1c011910 tx=0x7faa1c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.013+0000 7faa297fa640 1 -- 172.21.15.12:0/4247031492 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c005ba0 con 0x7faa180051c0 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.013+0000 7faa297fa640 1 -- 172.21.15.12:0/4247031492 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1c005d40 con 0x7faa180051c0 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.013+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2c161dd0 con 0x7faa180051c0 2024-04-15T15:21:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.013+0000 7faa297fa640 1 -- 172.21.15.12:0/4247031492 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c00fb00 con 0x7faa180051c0 2024-04-15T15:21:23.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.013+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2c161530 con 0x7faa180051c0 2024-04-15T15:21:23.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.014+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa2c0b7a90 con 0x7faa180051c0 2024-04-15T15:21:23.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.014+0000 7faa297fa640 1 -- 172.21.15.12:0/4247031492 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faa1c012040 con 0x7faa180051c0 2024-04-15T15:21:23.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.014+0000 7faa297fa640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa0c06c8f0 0x7faa0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.015+0000 7faa297fa640 1 -- 172.21.15.12:0/4247031492 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faa1c08c450 con 0x7faa180051c0 2024-04-15T15:21:23.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.015+0000 7faa23fff640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa0c06c8f0 0x7faa0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.015+0000 7faa23fff640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa0c06c8f0 0x7faa0c06edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faa2c1f7990 tx=0x7faa14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.019+0000 7faa297fa640 1 -- 172.21.15.12:0/4247031492 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa1c018030 con 0x7faa180051c0 2024-04-15T15:21:23.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.217+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7faa2c0689e0 con 0x7faa180051c0 2024-04-15T15:21:23.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.217+0000 7faa297fa640 1 -- 172.21.15.12:0/4247031492 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7faa1c01b360 con 0x7faa180051c0 2024-04-15T15:21:23.219 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:21:23.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa0c06c8f0 msgr2=0x7faa0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa0c06c8f0 0x7faa0c06edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faa2c1f7990 tx=0x7faa14009040 comp rx=0 tx=0).stop 2024-04-15T15:21:23.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 msgr2=0x7faa2c14b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa2c14b0a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa1c011910 tx=0x7faa1c011dd0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 shutdown_connections 2024-04-15T15:21:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa2c1608e0 0x7faa2c160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa0c06c8f0 0x7faa0c06edb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa2c1f7a30 0x7faa2c1f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 --2- 172.21.15.12:0/4247031492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa180051c0 0x7faa2c14b0a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.219+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 >> 172.21.15.12:0/4247031492 conn(0x7faa2c08bf60 msgr2=0x7faa2c0c7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.220+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 shutdown_connections 2024-04-15T15:21:23.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.220+0000 7faa32331640 1 -- 172.21.15.12:0/4247031492 wait complete. 2024-04-15T15:21:23.232 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:21:23.232 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:21:23.247 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:23.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.391+0000 7fa6cf959640 1 Processor -- start 2024-04-15T15:21:23.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.391+0000 7fa6cf959640 1 -- start start 2024-04-15T15:21:23.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cf959640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80d3480 0x7fa6c80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cf959640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6c80b6790 0x7fa6c80b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cf959640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6c80cff70 con 0x7fa6c815ea40 2024-04-15T15:21:23.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cf959640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa6c80d0250 con 0x7fa6c80b6790 2024-04-15T15:21:23.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cf959640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c80d0530 con 0x7fa6c80d3480 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cdef6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6c80b6790 0x7fa6c80b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cdef6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa6c80b6790 0x7fa6c80b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52848/0 (socket says 172.21.15.12:52848) 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cd6f5640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa6c815ea40 0x7fa6c80b5b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36454/0 (socket says 172.21.15.12:36454) 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cdef6640 1 -- 172.21.15.12:0/4077820439 learned_addr learned my addr 172.21.15.12:0/4077820439 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6ccef4640 1 --2- 172.21.15.12:0/4077820439 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80d3480 0x7fa6c80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cdef6640 1 -- 172.21.15.12:0/4077820439 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80d3480 msgr2=0x7fa6c80b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cdef6640 1 --2- 172.21.15.12:0/4077820439 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80d3480 0x7fa6c80b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cdef6640 1 -- 172.21.15.12:0/4077820439 >> v1:172.21.15.12:6789/0 conn(0x7fa6c815ea40 legacy=0x7fa6c80b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6bffff640 1 -- 172.21.15.12:0/4077820439 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088091287 0 0) 0x7fa6c80cff70 con 0x7fa6c815ea40 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6cdef6640 1 -- 172.21.15.12:0/4077820439 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c80d0810 con 0x7fa6c80b6790 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.392+0000 7fa6ccef4640 1 --2- 172.21.15.12:0/4077820439 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80d3480 0x7fa6c80b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.393+0000 7fa6cdef6640 1 --2- 172.21.15.12:0/4077820439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c80b6790 0x7fa6c80b3e80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009e50 tx=0x7fa6c002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b918716103dca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.393+0000 7fa6bffff640 1 -- 172.21.15.12:0/4077820439 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c002cb90 con 0x7fa6c80b6790 2024-04-15T15:21:23.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.393+0000 7fa6bffff640 1 -- 172.21.15.12:0/4077820439 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c002cd30 con 0x7fa6c80b6790 2024-04-15T15:21:23.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.393+0000 7fa6bffff640 1 -- 172.21.15.12:0/4077820439 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0034610 con 0x7fa6c80b6790 2024-04-15T15:21:23.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.393+0000 7fa6cf959640 1 -- 172.21.15.12:0/4077820439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c80b6790 msgr2=0x7fa6c80b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.393+0000 7fa6cf959640 1 --2- 172.21.15.12:0/4077820439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c80b6790 0x7fa6c80b3e80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009e50 tx=0x7fa6c002bff0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.394+0000 7fa6cf959640 1 -- 172.21.15.12:0/4077820439 shutdown_connections 2024-04-15T15:21:23.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.394+0000 7fa6cf959640 1 --2- 172.21.15.12:0/4077820439 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c80b6790 0x7fa6c80b3e80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.394+0000 7fa6cf959640 1 --2- 172.21.15.12:0/4077820439 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80d3480 0x7fa6c80b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.394+0000 7fa6cf959640 1 -- 172.21.15.12:0/4077820439 >> 172.21.15.12:0/4077820439 conn(0x7fa6c808bf60 msgr2=0x7fa6c80d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.394+0000 7fa6cf959640 1 -- 172.21.15.12:0/4077820439 shutdown_connections 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.394+0000 7fa6cf959640 1 -- 172.21.15.12:0/4077820439 wait complete. 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.395+0000 7fa6cf959640 1 Processor -- start 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.395+0000 7fa6cf959640 1 -- start start 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.395+0000 7fa6cf959640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80b6790 0x7fa6c8155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.395+0000 7fa6cf959640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6c80d3480 0x7fa6c80b8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cf959640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c815ea40 0x7fa6c80b8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cf959640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c80b3a40 con 0x7fa6c815ea40 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cf959640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c80b38c0 con 0x7fa6c80d3480 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cf959640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c815ee20 con 0x7fa6c80b6790 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c815ea40 0x7fa6c80b8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c815ea40 0x7fa6c80b8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52864/0 (socket says 172.21.15.12:52864) 2024-04-15T15:21:23.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 -- 172.21.15.12:0/3059376757 learned_addr learned my addr 172.21.15.12:0/3059376757 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cd6f5640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80b6790 0x7fa6c8155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6ccef4640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6c80d3480 0x7fa6c80b8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 -- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80b6790 msgr2=0x7fa6c8155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80b6790 0x7fa6c8155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 -- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6c80d3480 msgr2=0x7fa6c80b8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6c80d3480 0x7fa6c80b8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 -- 172.21.15.12:0/3059376757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c815f230 con 0x7fa6c815ea40 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cd6f5640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80b6790 0x7fa6c8155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6ccef4640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6c80d3480 0x7fa6c80b8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cdef6640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c815ea40 0x7fa6c80b8dd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0036040 tx=0x7fa6c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6bdffb640 1 -- 172.21.15.12:0/3059376757 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0043020 con 0x7fa6c815ea40 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6bdffb640 1 -- 172.21.15.12:0/3059376757 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c003e450 con 0x7fa6c815ea40 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c80b9560 con 0x7fa6c815ea40 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.396+0000 7fa6bdffb640 1 -- 172.21.15.12:0/3059376757 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0009510 con 0x7fa6c815ea40 2024-04-15T15:21:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.397+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c80b9b20 con 0x7fa6c815ea40 2024-04-15T15:21:23.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.397+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c80689e0 con 0x7fa6c815ea40 2024-04-15T15:21:23.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.398+0000 7fa6bdffb640 1 -- 172.21.15.12:0/3059376757 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa6c0046020 con 0x7fa6c815ea40 2024-04-15T15:21:23.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.398+0000 7fa6bdffb640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa6ac06c8f0 0x7fa6ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.399+0000 7fa6bdffb640 1 -- 172.21.15.12:0/3059376757 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa6c00ae590 con 0x7fa6c815ea40 2024-04-15T15:21:23.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.399+0000 7fa6cd6f5640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa6ac06c8f0 0x7fa6ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.399+0000 7fa6cd6f5640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa6ac06c8f0 0x7fa6ac06edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4000f60 tx=0x7fa6c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.403+0000 7fa6bdffb640 1 -- 172.21.15.12:0/3059376757 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6c004d030 con 0x7fa6c815ea40 2024-04-15T15:21:23.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.587+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6c80cf450 con 0x7fa6c815ea40 2024-04-15T15:21:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.588+0000 7fa6bdffb640 1 -- 172.21.15.12:0/3059376757 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7fa6c007c670 con 0x7fa6c815ea40 2024-04-15T15:21:23.590 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:23.590 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa6ac06c8f0 msgr2=0x7fa6ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa6ac06c8f0 0x7fa6ac06edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4000f60 tx=0x7fa6c4009040 comp rx=0 tx=0).stop 2024-04-15T15:21:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c815ea40 msgr2=0x7fa6c80b8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c815ea40 0x7fa6c80b8dd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0036040 tx=0x7fa6c0004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 shutdown_connections 2024-04-15T15:21:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6c815ea40 0x7fa6c80b8dd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa6ac06c8f0 0x7fa6ac06edb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6c80d3480 0x7fa6c80b8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 --2- 172.21.15.12:0/3059376757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa6c80b6790 0x7fa6c8155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.590+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 >> 172.21.15.12:0/3059376757 conn(0x7fa6c808bf60 msgr2=0x7fa6c806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.591+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 shutdown_connections 2024-04-15T15:21:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.591+0000 7fa6cf959640 1 -- 172.21.15.12:0/3059376757 wait complete. 2024-04-15T15:21:23.603 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:23.604 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.711+0000 7f9e45faf640 1 Processor -- start 2024-04-15T15:21:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.711+0000 7f9e45faf640 1 -- start start 2024-04-15T15:21:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e45faf640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 0x7f9e400b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e45faf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e400d3270 0x7f9e400cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e45faf640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e400b47a0 con 0x7f9e400d2360 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e45faf640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9e400b53b0 con 0x7f9e400d3270 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e45faf640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9e400b4a80 con 0x7f9e4015e700 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e3ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e400d3270 0x7f9e400cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e3ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e400d3270 0x7f9e400cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52870/0 (socket says 172.21.15.12:52870) 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e3ffff640 1 -- 172.21.15.12:0/574096011 learned_addr learned my addr 172.21.15.12:0/574096011 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.712+0000 7f9e3f7fe640 1 --2- 172.21.15.12:0/574096011 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 0x7f9e400b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.713+0000 7f9e3ffff640 1 -- 172.21.15.12:0/574096011 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 msgr2=0x7f9e400b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.713+0000 7f9e3ffff640 1 --2- 172.21.15.12:0/574096011 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 0x7f9e400b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.713+0000 7f9e3ffff640 1 -- 172.21.15.12:0/574096011 >> v1:172.21.15.55:6789/0 conn(0x7f9e400d2360 legacy=0x7f9e400d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.713+0000 7f9e3ffff640 1 -- 172.21.15.12:0/574096011 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e400b4d60 con 0x7f9e400d3270 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.713+0000 7f9e3ffff640 1 --2- 172.21.15.12:0/574096011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d3270 0x7f9e400cf370 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009e50 tx=0x7f9e3802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb97f04599e265ea server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.713+0000 7f9e3e7fc640 1 -- 172.21.15.12:0/574096011 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3802cb90 con 0x7f9e400d3270 2024-04-15T15:21:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.713+0000 7f9e3e7fc640 1 -- 172.21.15.12:0/574096011 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e3802cd30 con 0x7f9e400d3270 2024-04-15T15:21:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.713+0000 7f9e3e7fc640 1 -- 172.21.15.12:0/574096011 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38034610 con 0x7f9e400d3270 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.714+0000 7f9e45faf640 1 -- 172.21.15.12:0/574096011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d3270 msgr2=0x7f9e400cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.714+0000 7f9e45faf640 1 --2- 172.21.15.12:0/574096011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d3270 0x7f9e400cf370 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009e50 tx=0x7f9e3802bff0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.714+0000 7f9e45faf640 1 -- 172.21.15.12:0/574096011 shutdown_connections 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.714+0000 7f9e45faf640 1 --2- 172.21.15.12:0/574096011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d3270 0x7f9e400cf370 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.714+0000 7f9e45faf640 1 --2- 172.21.15.12:0/574096011 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 0x7f9e400b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.714+0000 7f9e45faf640 1 -- 172.21.15.12:0/574096011 >> 172.21.15.12:0/574096011 conn(0x7f9e4008bf60 msgr2=0x7f9e400d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.714+0000 7f9e45faf640 1 -- 172.21.15.12:0/574096011 shutdown_connections 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.714+0000 7f9e45faf640 1 -- 172.21.15.12:0/574096011 wait complete. 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.715+0000 7f9e45faf640 1 Processor -- start 2024-04-15T15:21:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.715+0000 7f9e45faf640 1 -- start start 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e45faf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d2360 0x7f9e40158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e45faf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9e400d3270 0x7f9e400b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e45faf640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 0x7f9e400b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e45faf640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e400b5d80 con 0x7f9e400d2360 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e45faf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9e400b4450 con 0x7f9e400d3270 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e45faf640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9e400d1d40 con 0x7f9e4015e700 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e3f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d2360 0x7f9e40158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e3f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d2360 0x7f9e40158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52886/0 (socket says 172.21.15.12:52886) 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.716+0000 7f9e3f7fe640 1 -- 172.21.15.12:0/1255229390 learned_addr learned my addr 172.21.15.12:0/1255229390 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3ffff640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 0x7f9e400b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3effd640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9e400d3270 0x7f9e400b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3f7fe640 1 -- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 msgr2=0x7f9e400b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3f7fe640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 0x7f9e400b91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3f7fe640 1 -- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9e400d3270 msgr2=0x7f9e400b8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3f7fe640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9e400d3270 0x7f9e400b8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3f7fe640 1 -- 172.21.15.12:0/1255229390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e40159330 con 0x7f9e400d2360 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3effd640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9e400d3270 0x7f9e400b8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e3f7fe640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d2360 0x7f9e40158df0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e3000e8a0 tx=0x7f9e3000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e27fff640 1 -- 172.21.15.12:0/1255229390 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30017070 con 0x7f9e400d2360 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e27fff640 1 -- 172.21.15.12:0/1255229390 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e3000c940 con 0x7f9e400d2360 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e400c9c70 con 0x7f9e400d2360 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e27fff640 1 -- 172.21.15.12:0/1255229390 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3000ccf0 con 0x7f9e400d2360 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.717+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e400c8eb0 con 0x7f9e400d2360 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.718+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e400690f0 con 0x7f9e400d2360 2024-04-15T15:21:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.719+0000 7f9e27fff640 1 -- 172.21.15.12:0/1255229390 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9e3001a070 con 0x7f9e400d2360 2024-04-15T15:21:23.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.719+0000 7f9e27fff640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9e2c06c8f0 0x7f9e2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:23.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.719+0000 7f9e27fff640 1 -- 172.21.15.12:0/1255229390 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9e3008c1f0 con 0x7f9e400d2360 2024-04-15T15:21:23.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.719+0000 7f9e3effd640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9e2c06c8f0 0x7f9e2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:23.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.720+0000 7f9e3effd640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9e2c06c8f0 0x7f9e2c06edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e40073620 tx=0x7f9e34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.723+0000 7f9e27fff640 1 -- 172.21.15.12:0/1255229390 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e400690f0 con 0x7f9e400d2360 2024-04-15T15:21:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.910+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9e400689e0 con 0x7f9e400d2360 2024-04-15T15:21:23.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.910+0000 7f9e27fff640 1 -- 172.21.15.12:0/1255229390 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f9e3005a2e0 con 0x7f9e400d2360 2024-04-15T15:21:23.912 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:23.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9e2c06c8f0 msgr2=0x7f9e2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9e2c06c8f0 0x7f9e2c06edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e40073620 tx=0x7f9e34009040 comp rx=0 tx=0).stop 2024-04-15T15:21:23.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d2360 msgr2=0x7f9e40158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:23.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d2360 0x7f9e40158df0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e3000e8a0 tx=0x7f9e3000ed60 comp rx=0 tx=0).stop 2024-04-15T15:21:23.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 shutdown_connections 2024-04-15T15:21:23.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9e4015e700 0x7f9e400b91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9e2c06c8f0 0x7f9e2c06edb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9e400d3270 0x7f9e400b8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 --2- 172.21.15.12:0/1255229390 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e400d2360 0x7f9e40158df0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 >> 172.21.15.12:0/1255229390 conn(0x7f9e4008bf60 msgr2=0x7f9e400d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 shutdown_connections 2024-04-15T15:21:23.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:23.912+0000 7f9e45faf640 1 -- 172.21.15.12:0/1255229390 wait complete. 2024-04-15T15:21:23.925 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.031+0000 7fdac4fba640 1 Processor -- start 2024-04-15T15:21:24.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.031+0000 7fdac4fba640 1 -- start start 2024-04-15T15:21:24.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdac4fba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdac015e700 0x7fdac00b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdac4fba640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdac00d2360 0x7fdac00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdac4fba640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac00b47a0 con 0x7fdac015eae0 2024-04-15T15:21:24.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdac4fba640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fdac00b53b0 con 0x7fdac00d2360 2024-04-15T15:21:24.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdac4fba640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fdac00b4a80 con 0x7fdac015e700 2024-04-15T15:21:24.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdab7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdac00d2360 0x7fdac00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdab7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdac00d2360 0x7fdac00d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52896/0 (socket says 172.21.15.12:52896) 2024-04-15T15:21:24.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdab7fff640 1 -- 172.21.15.12:0/3498450178 learned_addr learned my addr 172.21.15.12:0/3498450178 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdabe59c640 1 --2- 172.21.15.12:0/3498450178 >> v2:172.21.15.55:3300/0 conn(0x7fdac015e700 0x7fdac00b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdab7fff640 1 -- 172.21.15.12:0/3498450178 >> v2:172.21.15.55:3300/0 conn(0x7fdac015e700 msgr2=0x7fdac00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdab7fff640 1 --2- 172.21.15.12:0/3498450178 >> v2:172.21.15.55:3300/0 conn(0x7fdac015e700 0x7fdac00b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdab7fff640 1 -- 172.21.15.12:0/3498450178 >> v1:172.21.15.55:6789/0 conn(0x7fdac015eae0 legacy=0x7fdac00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.032+0000 7fdab7fff640 1 -- 172.21.15.12:0/3498450178 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac00b4d60 con 0x7fdac00d2360 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.033+0000 7fdab7fff640 1 --2- 172.21.15.12:0/3498450178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac00d2360 0x7fdac00d2b60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdac006c6c0 tx=0x7fdaa802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0699030234f93bb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.033+0000 7fdabdd9b640 1 -- 172.21.15.12:0/3498450178 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8035020 con 0x7fdac00d2360 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.033+0000 7fdabdd9b640 1 -- 172.21.15.12:0/3498450178 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa8031870 con 0x7fdac00d2360 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.033+0000 7fdabdd9b640 1 -- 172.21.15.12:0/3498450178 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8031c40 con 0x7fdac00d2360 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.034+0000 7fdac4fba640 1 -- 172.21.15.12:0/3498450178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac00d2360 msgr2=0x7fdac00d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.034+0000 7fdac4fba640 1 --2- 172.21.15.12:0/3498450178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac00d2360 0x7fdac00d2b60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdac006c6c0 tx=0x7fdaa802b910 comp rx=0 tx=0).stop 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.034+0000 7fdac4fba640 1 -- 172.21.15.12:0/3498450178 shutdown_connections 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.034+0000 7fdac4fba640 1 --2- 172.21.15.12:0/3498450178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac00d2360 0x7fdac00d2b60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.034+0000 7fdac4fba640 1 --2- 172.21.15.12:0/3498450178 >> v2:172.21.15.55:3300/0 conn(0x7fdac015e700 0x7fdac00b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.034+0000 7fdac4fba640 1 -- 172.21.15.12:0/3498450178 >> 172.21.15.12:0/3498450178 conn(0x7fdac008bf60 msgr2=0x7fdac00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:24.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.034+0000 7fdac4fba640 1 -- 172.21.15.12:0/3498450178 shutdown_connections 2024-04-15T15:21:24.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.034+0000 7fdac4fba640 1 -- 172.21.15.12:0/3498450178 wait complete. 2024-04-15T15:21:24.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.035+0000 7fdac4fba640 1 Processor -- start 2024-04-15T15:21:24.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.035+0000 7fdac4fba640 1 -- start start 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdac4fba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdac00d2360 0x7fdac00c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdac4fba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac015e700 0x7fdac00cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdac4fba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdac015eae0 0x7fdac0150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdac4fba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdac00b5d80 con 0x7fdac015e700 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdac4fba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdac00b4450 con 0x7fdac00d2360 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdac4fba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fdac00d1d40 con 0x7fdac015eae0 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac015e700 0x7fdac00cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac015e700 0x7fdac00cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52910/0 (socket says 172.21.15.12:52910) 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 -- 172.21.15.12:0/1854776221 learned_addr learned my addr 172.21.15.12:0/1854776221 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdabe59c640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdac00d2360 0x7fdac00c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdabed9d640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdac015eae0 0x7fdac0150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 -- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdac015eae0 msgr2=0x7fdac0150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdac015eae0 0x7fdac0150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 -- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdac00d2360 msgr2=0x7fdac00c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdac00d2360 0x7fdac00c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 -- 172.21.15.12:0/1854776221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac014e1d0 con 0x7fdac015e700 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.036+0000 7fdab7fff640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac015e700 0x7fdac00cdba0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8000c00 tx=0x7fdaa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.037+0000 7fdab6ffd640 1 -- 172.21.15.12:0/1854776221 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8040070 con 0x7fdac015e700 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.037+0000 7fdab6ffd640 1 -- 172.21.15.12:0/1854776221 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa8030d00 con 0x7fdac015e700 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.037+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac014e4b0 con 0x7fdac015e700 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.037+0000 7fdab6ffd640 1 -- 172.21.15.12:0/1854776221 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8035040 con 0x7fdac015e700 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.037+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac014ea70 con 0x7fdac015e700 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.038+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac00689e0 con 0x7fdac015e700 2024-04-15T15:21:24.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.038+0000 7fdab6ffd640 1 -- 172.21.15.12:0/1854776221 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdaa804b060 con 0x7fdac015e700 2024-04-15T15:21:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.039+0000 7fdab6ffd640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaa006c8f0 0x7fdaa006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.039+0000 7fdab6ffd640 1 -- 172.21.15.12:0/1854776221 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdaa80adc80 con 0x7fdac015e700 2024-04-15T15:21:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.039+0000 7fdabe59c640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaa006c8f0 0x7fdaa006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.039+0000 7fdabe59c640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaa006c8f0 0x7fdaa006edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdab000b9b0 tx=0x7fdab0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.043+0000 7fdab6ffd640 1 -- 172.21.15.12:0/1854776221 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdaa807bd70 con 0x7fdac015e700 2024-04-15T15:21:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.230+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdac00ba500 con 0x7fdac015e700 2024-04-15T15:21:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.230+0000 7fdab6ffd640 1 -- 172.21.15.12:0/1854776221 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fdaa807b670 con 0x7fdac015e700 2024-04-15T15:21:24.232 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaa006c8f0 msgr2=0x7fdaa006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaa006c8f0 0x7fdaa006edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdab000b9b0 tx=0x7fdab0008040 comp rx=0 tx=0).stop 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac015e700 msgr2=0x7fdac00cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac015e700 0x7fdac00cdba0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8000c00 tx=0x7fdaa80046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 shutdown_connections 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdac015eae0 0x7fdac0150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac015e700 0x7fdac00cdba0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaa006c8f0 0x7fdaa006edb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 --2- 172.21.15.12:0/1854776221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdac00d2360 0x7fdac00c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 >> 172.21.15.12:0/1854776221 conn(0x7fdac008bf60 msgr2=0x7fdac00d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 shutdown_connections 2024-04-15T15:21:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.232+0000 7fdac4fba640 1 -- 172.21.15.12:0/1854776221 wait complete. 2024-04-15T15:21:24.244 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:24.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.350+0000 7f55107b5640 1 Processor -- start 2024-04-15T15:21:24.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.350+0000 7f55107b5640 1 -- start start 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f55107b5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f550815e700 0x7f55080b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f55107b5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55080d2360 0x7f55080d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f55107b5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55080b47a0 con 0x7f550815eae0 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f55107b5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f55080b53b0 con 0x7f550815e700 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f55107b5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f55080b4a80 con 0x7f55080d2360 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f550ed52640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f550815eae0 0x7f55080cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36464/0 (socket says 172.21.15.12:36464) 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f550ed52640 1 -- 172.21.15.12:0/1009355951 learned_addr learned my addr 172.21.15.12:0/1009355951 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f550e551640 1 --2- 172.21.15.12:0/1009355951 >> v2:172.21.15.55:3300/0 conn(0x7f550815e700 0x7f55080b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.351+0000 7f550dd50640 1 --2- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55080d2360 0x7f55080d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.352+0000 7f550d54f640 1 -- 172.21.15.12:0/1009355951 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3871107821 0 0) 0x7f55080b47a0 con 0x7f550815eae0 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.352+0000 7f550d54f640 1 -- 172.21.15.12:0/1009355951 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54f0003520 con 0x7f550815eae0 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.352+0000 7f550e551640 1 -- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55080d2360 msgr2=0x7f55080d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.352+0000 7f550e551640 1 --2- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55080d2360 0x7f55080d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.352+0000 7f550e551640 1 -- 172.21.15.12:0/1009355951 >> v1:172.21.15.12:6789/0 conn(0x7f550815eae0 legacy=0x7f55080cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.352+0000 7f550d54f640 1 -- 172.21.15.12:0/1009355951 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 445015495 0 0) 0x7f54f0003520 con 0x7f550815eae0 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.352+0000 7f550e551640 1 -- 172.21.15.12:0/1009355951 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55080b4d60 con 0x7f550815e700 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.352+0000 7f550dd50640 1 --2- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55080d2360 0x7f55080d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f550e551640 1 --2- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 0x7f55080b5840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009de0 tx=0x7f54f802bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8a9b07d1660bbec server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f550d54f640 1 -- 172.21.15.12:0/1009355951 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f8039070 con 0x7f550815e700 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f550d54f640 1 -- 172.21.15.12:0/1009355951 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f802c8e0 con 0x7f550815e700 2024-04-15T15:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f550d54f640 1 -- 172.21.15.12:0/1009355951 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500001670 con 0x7f550815e700 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f55107b5640 1 -- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 msgr2=0x7f55080b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f55107b5640 1 --2- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 0x7f55080b5840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009de0 tx=0x7f54f802bdb0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f55107b5640 1 -- 172.21.15.12:0/1009355951 shutdown_connections 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f55107b5640 1 --2- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55080d2360 0x7f55080d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f55107b5640 1 --2- 172.21.15.12:0/1009355951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 0x7f55080b5840 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f55107b5640 1 -- 172.21.15.12:0/1009355951 >> 172.21.15.12:0/1009355951 conn(0x7f550808bf60 msgr2=0x7f55080d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f55107b5640 1 -- 172.21.15.12:0/1009355951 shutdown_connections 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.353+0000 7f55107b5640 1 -- 172.21.15.12:0/1009355951 wait complete. 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f55107b5640 1 Processor -- start 2024-04-15T15:21:24.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f55107b5640 1 -- start start 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f55107b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55080d2360 0x7f55080b9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f55107b5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 0x7f5508152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f55107b5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f550815eae0 0x7f5508160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f55107b5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f55080b5d80 con 0x7f55080d2360 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f55107b5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f55080b4450 con 0x7f550815e700 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f55107b5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f55080d1d40 con 0x7f550815eae0 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f550e551640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55080d2360 0x7f55080b9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f550e551640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55080d2360 0x7f55080b9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52924/0 (socket says 172.21.15.12:52924) 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.355+0000 7f550e551640 1 -- 172.21.15.12:0/4265664153 learned_addr learned my addr 172.21.15.12:0/4265664153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f550dd50640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 0x7f5508152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f550ed52640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f550815eae0 0x7f5508160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f550e551640 1 -- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f550815eae0 msgr2=0x7f5508160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f550e551640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f550815eae0 0x7f5508160a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f550e551640 1 -- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 msgr2=0x7f5508152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f550e551640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 0x7f5508152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f550e551640 1 -- 172.21.15.12:0/4265664153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55080b8af0 con 0x7f55080d2360 2024-04-15T15:21:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f550e551640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55080d2360 0x7f55080b9b50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f54f8037040 tx=0x7f54f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f5506ffd640 1 -- 172.21.15.12:0/4265664153 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f8039050 con 0x7f55080d2360 2024-04-15T15:21:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f5506ffd640 1 -- 172.21.15.12:0/4265664153 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f8045070 con 0x7f55080d2360 2024-04-15T15:21:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55080b8d90 con 0x7f55080d2360 2024-04-15T15:21:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f5506ffd640 1 -- 172.21.15.12:0/4265664153 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f8009450 con 0x7f55080d2360 2024-04-15T15:21:24.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.356+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55080b9350 con 0x7f55080d2360 2024-04-15T15:21:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.357+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55080689e0 con 0x7f55080d2360 2024-04-15T15:21:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.358+0000 7f5506ffd640 1 -- 172.21.15.12:0/4265664153 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f54f8040690 con 0x7f55080d2360 2024-04-15T15:21:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.358+0000 7f5506ffd640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f54f406c8f0 0x7f54f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.358+0000 7f5506ffd640 1 -- 172.21.15.12:0/4265664153 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f54f80ae690 con 0x7f55080d2360 2024-04-15T15:21:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.358+0000 7f550dd50640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f54f406c8f0 0x7f54f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.359+0000 7f550dd50640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f54f406c8f0 0x7f54f406edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f54fc009560 tx=0x7f54fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.363+0000 7f5506ffd640 1 -- 172.21.15.12:0/4265664153 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54f807c810 con 0x7f55080d2360 2024-04-15T15:21:24.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.546+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f55080ba500 con 0x7f55080d2360 2024-04-15T15:21:24.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.547+0000 7f5506ffd640 1 -- 172.21.15.12:0/4265664153 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f54f807c110 con 0x7f55080d2360 2024-04-15T15:21:24.548 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:24.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.548+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f54f406c8f0 msgr2=0x7f54f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.548+0000 7f55107b5640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f54f406c8f0 0x7f54f406edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f54fc009560 tx=0x7f54fc008040 comp rx=0 tx=0).stop 2024-04-15T15:21:24.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.548+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55080d2360 msgr2=0x7f55080b9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.548+0000 7f55107b5640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55080d2360 0x7f55080b9b50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f54f8037040 tx=0x7f54f8004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:24.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.549+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 shutdown_connections 2024-04-15T15:21:24.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.549+0000 7f55107b5640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f550815eae0 0x7f5508160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.549+0000 7f55107b5640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f54f406c8f0 0x7f54f406edb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.549+0000 7f55107b5640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f550815e700 0x7f5508152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.549+0000 7f55107b5640 1 --2- 172.21.15.12:0/4265664153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55080d2360 0x7f55080b9b50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.549+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 >> 172.21.15.12:0/4265664153 conn(0x7f550808bf60 msgr2=0x7f55080d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:24.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.549+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 shutdown_connections 2024-04-15T15:21:24.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.549+0000 7f55107b5640 1 -- 172.21.15.12:0/4265664153 wait complete. 2024-04-15T15:21:24.561 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:24.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.667+0000 7ff253d0e640 1 Processor -- start 2024-04-15T15:21:24.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.668+0000 7ff253d0e640 1 -- start start 2024-04-15T15:21:24.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.668+0000 7ff253d0e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff24c0cf600 0x7ff24c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.668+0000 7ff253d0e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff24c0ae440 con 0x7ff24c15ed00 2024-04-15T15:21:24.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.668+0000 7ff253d0e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff24c0ae720 con 0x7ff24c0af4e0 2024-04-15T15:21:24.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.668+0000 7ff253d0e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff24c071030 con 0x7ff24c0cf600 2024-04-15T15:21:24.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.668+0000 7ff2512a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff24c0cf600 0x7ff24c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.668+0000 7ff251aaa640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff24c15ed00 0x7ff24c070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36468/0 (socket says 172.21.15.12:36468) 2024-04-15T15:21:24.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.668+0000 7ff251aaa640 1 -- 172.21.15.12:0/3295151257 learned_addr learned my addr 172.21.15.12:0/3295151257 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:24.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.669+0000 7ff2512a9640 1 -- 172.21.15.12:0/3295151257 >> v1:172.21.15.55:6789/0 conn(0x7ff24c0af4e0 legacy=0x7ff24c0af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.669+0000 7ff2512a9640 1 -- 172.21.15.12:0/3295151257 >> v1:172.21.15.12:6789/0 conn(0x7ff24c15ed00 legacy=0x7ff24c070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.669+0000 7ff2512a9640 1 -- 172.21.15.12:0/3295151257 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c0aea00 con 0x7ff24c0cf600 2024-04-15T15:21:24.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.669+0000 7ff250aa8640 1 -- 172.21.15.12:0/3295151257 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2245958480 0 0) 0x7ff24c0ae440 con 0x7ff24c15ed00 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff2512a9640 1 --2- 172.21.15.12:0/3295151257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c0cf600 0x7ff24c0cfa90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff2400097a0 tx=0x7ff24002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b129c6eb9701da3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff250aa8640 1 -- 172.21.15.12:0/3295151257 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240034070 con 0x7ff24c0cf600 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff250aa8640 1 -- 172.21.15.12:0/3295151257 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff240030610 con 0x7ff24c0cf600 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff250aa8640 1 -- 172.21.15.12:0/3295151257 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2400309e0 con 0x7ff24c0cf600 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff253d0e640 1 -- 172.21.15.12:0/3295151257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c0cf600 msgr2=0x7ff24c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff253d0e640 1 --2- 172.21.15.12:0/3295151257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c0cf600 0x7ff24c0cfa90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff2400097a0 tx=0x7ff24002b960 comp rx=0 tx=0).stop 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff253d0e640 1 -- 172.21.15.12:0/3295151257 shutdown_connections 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff253d0e640 1 --2- 172.21.15.12:0/3295151257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c0cf600 0x7ff24c0cfa90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff253d0e640 1 -- 172.21.15.12:0/3295151257 >> 172.21.15.12:0/3295151257 conn(0x7ff24c09a9d0 msgr2=0x7ff24c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:24.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff253d0e640 1 -- 172.21.15.12:0/3295151257 shutdown_connections 2024-04-15T15:21:24.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.670+0000 7ff253d0e640 1 -- 172.21.15.12:0/3295151257 wait complete. 2024-04-15T15:21:24.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.671+0000 7ff253d0e640 1 Processor -- start 2024-04-15T15:21:24.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff253d0e640 1 -- start start 2024-04-15T15:21:24.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff253d0e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff24c0af4e0 0x7ff24c14fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff253d0e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff24c0cf600 0x7ff24c0b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff253d0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c15ed00 0x7ff24c0caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff253d0e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c0701a0 con 0x7ff24c15ed00 2024-04-15T15:21:24.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff253d0e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c0aefc0 con 0x7ff24c0cf600 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff253d0e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff24c0b0020 con 0x7ff24c0af4e0 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff2522ab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c15ed00 0x7ff24c0caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff2522ab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c15ed00 0x7ff24c0caf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52938/0 (socket says 172.21.15.12:52938) 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff2522ab640 1 -- 172.21.15.12:0/3692500350 learned_addr learned my addr 172.21.15.12:0/3692500350 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff251aaa640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff24c0af4e0 0x7ff24c14fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.672+0000 7ff2512a9640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff24c0cf600 0x7ff24c0b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff2522ab640 1 -- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff24c0af4e0 msgr2=0x7ff24c14fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff2522ab640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff24c0af4e0 0x7ff24c14fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff2522ab640 1 -- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff24c0cf600 msgr2=0x7ff24c0b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff2522ab640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff24c0cf600 0x7ff24c0b6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff2522ab640 1 -- 172.21.15.12:0/3692500350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c0ca250 con 0x7ff24c15ed00 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff2522ab640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c15ed00 0x7ff24c0caf80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00c7a0 tx=0x7ff23c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff24a7fc640 1 -- 172.21.15.12:0/3692500350 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c003b70 con 0x7ff24c15ed00 2024-04-15T15:21:24.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff24a7fc640 1 -- 172.21.15.12:0/3692500350 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23c003d30 con 0x7ff24c15ed00 2024-04-15T15:21:24.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24c0c50c0 con 0x7ff24c15ed00 2024-04-15T15:21:24.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff24a7fc640 1 -- 172.21.15.12:0/3692500350 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c00aa90 con 0x7ff24c15ed00 2024-04-15T15:21:24.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.673+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24c0ca780 con 0x7ff24c15ed00 2024-04-15T15:21:24.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.674+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff24c0690f0 con 0x7ff24c15ed00 2024-04-15T15:21:24.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.674+0000 7ff24a7fc640 1 -- 172.21.15.12:0/3692500350 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff23c00ac30 con 0x7ff24c15ed00 2024-04-15T15:21:24.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.675+0000 7ff24a7fc640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff23806c8f0 0x7ff23806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.675+0000 7ff24a7fc640 1 -- 172.21.15.12:0/3692500350 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff23c08bcc0 con 0x7ff24c15ed00 2024-04-15T15:21:24.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.675+0000 7ff251aaa640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff23806c8f0 0x7ff23806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.676+0000 7ff251aaa640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff23806c8f0 0x7ff23806edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff244000f60 tx=0x7ff244007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.679+0000 7ff24a7fc640 1 -- 172.21.15.12:0/3692500350 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff23c0596e0 con 0x7ff24c15ed00 2024-04-15T15:21:24.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.865+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff24c0ca560 con 0x7ff24c15ed00 2024-04-15T15:21:24.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.865+0000 7ff24a7fc640 1 -- 172.21.15.12:0/3692500350 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7ff23c05d9e0 con 0x7ff24c15ed00 2024-04-15T15:21:24.867 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff23806c8f0 msgr2=0x7ff23806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff23806c8f0 0x7ff23806edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff244000f60 tx=0x7ff244007e40 comp rx=0 tx=0).stop 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c15ed00 msgr2=0x7ff24c0caf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c15ed00 0x7ff24c0caf80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00c7a0 tx=0x7ff23c00cc60 comp rx=0 tx=0).stop 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 shutdown_connections 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff24c15ed00 0x7ff24c0caf80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff23806c8f0 0x7ff23806edb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff24c0cf600 0x7ff24c0b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 --2- 172.21.15.12:0/3692500350 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff24c0af4e0 0x7ff24c14fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 >> 172.21.15.12:0/3692500350 conn(0x7ff24c09a9d0 msgr2=0x7ff24c0d1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 shutdown_connections 2024-04-15T15:21:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.867+0000 7ff253d0e640 1 -- 172.21.15.12:0/3692500350 wait complete. 2024-04-15T15:21:24.880 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:21:24.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.984+0000 7f1e63242640 1 Processor -- start 2024-04-15T15:21:24.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.984+0000 7f1e63242640 1 -- start start 2024-04-15T15:21:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.984+0000 7f1e63242640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e5c070940 0x7f1e5c09b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.984+0000 7f1e63242640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1e5c160310 0x7f1e5c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.984+0000 7f1e63242640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e5c0d05e0 con 0x7f1e5c0d0da0 2024-04-15T15:21:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.984+0000 7f1e63242640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1e5c071260 con 0x7f1e5c070940 2024-04-15T15:21:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.984+0000 7f1e63242640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1e5c15f720 con 0x7f1e5c160310 2024-04-15T15:21:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e60fde640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e5c070940 0x7f1e5c09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e617df640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1e5c0d0da0 0x7f1e5c0d11d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36472/0 (socket says 172.21.15.12:36472) 2024-04-15T15:21:24.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e60fde640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e5c070940 0x7f1e5c09b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52948/0 (socket says 172.21.15.12:52948) 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e617df640 1 -- 172.21.15.12:0/3333753480 learned_addr learned my addr 172.21.15.12:0/3333753480 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e5bfff640 1 --2- 172.21.15.12:0/3333753480 >> v2:172.21.15.55:3300/0 conn(0x7f1e5c160310 0x7f1e5c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e60fde640 1 -- 172.21.15.12:0/3333753480 >> v2:172.21.15.55:3300/0 conn(0x7f1e5c160310 msgr2=0x7f1e5c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e60fde640 1 --2- 172.21.15.12:0/3333753480 >> v2:172.21.15.55:3300/0 conn(0x7f1e5c160310 0x7f1e5c070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e60fde640 1 -- 172.21.15.12:0/3333753480 >> v1:172.21.15.12:6789/0 conn(0x7f1e5c0d0da0 legacy=0x7f1e5c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e60fde640 1 -- 172.21.15.12:0/3333753480 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c15fa00 con 0x7f1e5c070940 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.985+0000 7f1e5b7fe640 1 -- 172.21.15.12:0/3333753480 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050089079 0 0) 0x7f1e5c0d05e0 con 0x7f1e5c0d0da0 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e60fde640 1 --2- 172.21.15.12:0/3333753480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c070940 0x7f1e5c09b880 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0734f0 tx=0x7f1e5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2286e07903b11b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e5b7fe640 1 -- 172.21.15.12:0/3333753480 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50035020 con 0x7f1e5c070940 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e5b7fe640 1 -- 172.21.15.12:0/3333753480 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e50031830 con 0x7f1e5c070940 2024-04-15T15:21:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e5b7fe640 1 -- 172.21.15.12:0/3333753480 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50031c00 con 0x7f1e5c070940 2024-04-15T15:21:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e63242640 1 -- 172.21.15.12:0/3333753480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c070940 msgr2=0x7f1e5c09b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e63242640 1 --2- 172.21.15.12:0/3333753480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c070940 0x7f1e5c09b880 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0734f0 tx=0x7f1e5002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e63242640 1 -- 172.21.15.12:0/3333753480 shutdown_connections 2024-04-15T15:21:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e63242640 1 --2- 172.21.15.12:0/3333753480 >> v2:172.21.15.55:3300/0 conn(0x7f1e5c160310 0x7f1e5c070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e63242640 1 --2- 172.21.15.12:0/3333753480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c070940 0x7f1e5c09b880 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.986+0000 7f1e63242640 1 -- 172.21.15.12:0/3333753480 >> 172.21.15.12:0/3333753480 conn(0x7f1e5c0b3f40 msgr2=0x7f1e5c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.987+0000 7f1e63242640 1 -- 172.21.15.12:0/3333753480 shutdown_connections 2024-04-15T15:21:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.987+0000 7f1e63242640 1 -- 172.21.15.12:0/3333753480 wait complete. 2024-04-15T15:21:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.988+0000 7f1e63242640 1 Processor -- start 2024-04-15T15:21:24.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.988+0000 7f1e63242640 1 -- start start 2024-04-15T15:21:24.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.988+0000 7f1e63242640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e5c070940 0x7f1e5c0c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e63242640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e5c0d0da0 0x7f1e5c1f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e63242640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c160310 0x7f1e5c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e63242640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c0d18e0 con 0x7f1e5c160310 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e63242640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c0d0880 con 0x7f1e5c070940 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e63242640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1e5c160770 con 0x7f1e5c0d0da0 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c160310 0x7f1e5c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c160310 0x7f1e5c06db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52950/0 (socket says 172.21.15.12:52950) 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 -- 172.21.15.12:0/755467507 learned_addr learned my addr 172.21.15.12:0/755467507 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e60fde640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e5c070940 0x7f1e5c0c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 -- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e5c0d0da0 msgr2=0x7f1e5c1f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e5c0d0da0 0x7f1e5c1f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 -- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e5c070940 msgr2=0x7f1e5c0c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e5c070940 0x7f1e5c0c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 -- 172.21.15.12:0/755467507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c0c6370 con 0x7f1e5c160310 2024-04-15T15:21:24.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e617df640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c160310 0x7f1e5c06db00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400c7b0 tx=0x7f1e5400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e597fa640 1 -- 172.21.15.12:0/755467507 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54018070 con 0x7f1e5c160310 2024-04-15T15:21:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e597fa640 1 -- 172.21.15.12:0/755467507 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e5400a640 con 0x7f1e5c160310 2024-04-15T15:21:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5c0c6610 con 0x7f1e5c160310 2024-04-15T15:21:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e597fa640 1 -- 172.21.15.12:0/755467507 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5400aa30 con 0x7f1e5c160310 2024-04-15T15:21:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.989+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5c0c6bd0 con 0x7f1e5c160310 2024-04-15T15:21:24.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.990+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e5c0689e0 con 0x7f1e5c160310 2024-04-15T15:21:24.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.991+0000 7f1e597fa640 1 -- 172.21.15.12:0/755467507 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e5401b070 con 0x7f1e5c160310 2024-04-15T15:21:24.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.991+0000 7f1e597fa640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e4006c8f0 0x7f1e4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:24.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.992+0000 7f1e597fa640 1 -- 172.21.15.12:0/755467507 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1e5408b3f0 con 0x7f1e5c160310 2024-04-15T15:21:24.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.992+0000 7f1e60fde640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e4006c8f0 0x7f1e4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:24.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.992+0000 7f1e60fde640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e4006c8f0 0x7f1e4006edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1e50036000 tx=0x7f1e50008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:24.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:24.996+0000 7f1e597fa640 1 -- 172.21.15.12:0/755467507 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e54059570 con 0x7f1e5c160310 2024-04-15T15:21:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.197+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1e5c070720 con 0x7f1e5c160310 2024-04-15T15:21:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.197+0000 7f1e597fa640 1 -- 172.21.15.12:0/755467507 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f1e54058e70 con 0x7f1e5c160310 2024-04-15T15:21:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e4006c8f0 msgr2=0x7f1e4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e4006c8f0 0x7f1e4006edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1e50036000 tx=0x7f1e50008030 comp rx=0 tx=0).stop 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c160310 msgr2=0x7f1e5c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c160310 0x7f1e5c06db00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400c7b0 tx=0x7f1e5400cc70 comp rx=0 tx=0).stop 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 shutdown_connections 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e5c160310 0x7f1e5c06db00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e5c0d0da0 0x7f1e5c1f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e4006c8f0 0x7f1e4006edb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 --2- 172.21.15.12:0/755467507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e5c070940 0x7f1e5c0c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 >> 172.21.15.12:0/755467507 conn(0x7f1e5c0b3f40 msgr2=0x7f1e5c15ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 shutdown_connections 2024-04-15T15:21:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:25.199+0000 7f1e63242640 1 -- 172.21.15.12:0/755467507 wait complete. 2024-04-15T15:21:25.211 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-04-15T15:21:25.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:25.212 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:21:25.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[44481]: fuse finished with error 0 and tester_r 0 2024-04-15T15:21:25.239 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:21:31.341 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:21:31.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:31.342 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-15T15:21:31.370 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:31.405 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:21:31.413 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:21:31.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.525+0000 7f480e97c640 1 Processor -- start 2024-04-15T15:21:31.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.526+0000 7f480e97c640 1 -- start start 2024-04-15T15:21:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.526+0000 7f480e97c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f480815e700 0x7f48080b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.526+0000 7f480e97c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f48080d2360 0x7f48080d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.526+0000 7f480e97c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48080b47a0 con 0x7f480815eae0 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.526+0000 7f480e97c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f48080b53b0 con 0x7f48080d2360 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.526+0000 7f480e97c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f48080b4a80 con 0x7f480815e700 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f48077fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f48080d2360 0x7f48080d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f48077fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f48080d2360 0x7f48080d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52962/0 (socket says 172.21.15.12:52962) 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f480cf19640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f480815eae0 0x7f48080cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36476/0 (socket says 172.21.15.12:36476) 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f48077fe640 1 -- 172.21.15.12:0/2833518296 learned_addr learned my addr 172.21.15.12:0/2833518296 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f4807fff640 1 --2- 172.21.15.12:0/2833518296 >> v2:172.21.15.55:3300/0 conn(0x7f480815e700 0x7f48080b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f48077fe640 1 -- 172.21.15.12:0/2833518296 >> v2:172.21.15.55:3300/0 conn(0x7f480815e700 msgr2=0x7f48080b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f48077fe640 1 --2- 172.21.15.12:0/2833518296 >> v2:172.21.15.55:3300/0 conn(0x7f480815e700 0x7f48080b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f48077fe640 1 -- 172.21.15.12:0/2833518296 >> v1:172.21.15.12:6789/0 conn(0x7f480815eae0 legacy=0x7f48080cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f48077fe640 1 -- 172.21.15.12:0/2833518296 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48080b4d60 con 0x7f48080d2360 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f4806ffd640 1 -- 172.21.15.12:0/2833518296 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646638046 0 0) 0x7f48080b47a0 con 0x7f480815eae0 2024-04-15T15:21:31.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.527+0000 7f4807fff640 1 --2- 172.21.15.12:0/2833518296 >> v2:172.21.15.55:3300/0 conn(0x7f480815e700 0x7f48080b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f48077fe640 1 --2- 172.21.15.12:0/2833518296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48080d2360 0x7f48080d2b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f480806c6c0 tx=0x7f47fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea8d71522df425a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f4806ffd640 1 -- 172.21.15.12:0/2833518296 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc034070 con 0x7f48080d2360 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f4806ffd640 1 -- 172.21.15.12:0/2833518296 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47fc030860 con 0x7f48080d2360 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f4806ffd640 1 -- 172.21.15.12:0/2833518296 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc030c30 con 0x7f48080d2360 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f480e97c640 1 -- 172.21.15.12:0/2833518296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48080d2360 msgr2=0x7f48080d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f480e97c640 1 --2- 172.21.15.12:0/2833518296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48080d2360 0x7f48080d2b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f480806c6c0 tx=0x7f47fc02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f480e97c640 1 -- 172.21.15.12:0/2833518296 shutdown_connections 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f480e97c640 1 --2- 172.21.15.12:0/2833518296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48080d2360 0x7f48080d2b60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f480e97c640 1 --2- 172.21.15.12:0/2833518296 >> v2:172.21.15.55:3300/0 conn(0x7f480815e700 0x7f48080b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f480e97c640 1 -- 172.21.15.12:0/2833518296 >> 172.21.15.12:0/2833518296 conn(0x7f480808bf60 msgr2=0x7f48080d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f480e97c640 1 -- 172.21.15.12:0/2833518296 shutdown_connections 2024-04-15T15:21:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.528+0000 7f480e97c640 1 -- 172.21.15.12:0/2833518296 wait complete. 2024-04-15T15:21:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.530+0000 7f480e97c640 1 Processor -- start 2024-04-15T15:21:31.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.530+0000 7f480e97c640 1 -- start start 2024-04-15T15:21:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480e97c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f48080d2360 0x7f48081f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480e97c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f480815e700 0x7f4808153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480e97c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f480815eae0 0x7f48080cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480e97c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f48080b5d80 con 0x7f480815eae0 2024-04-15T15:21:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480e97c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f48080b4450 con 0x7f480815e700 2024-04-15T15:21:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480e97c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f48080d1d40 con 0x7f48080d2360 2024-04-15T15:21:31.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f480815eae0 0x7f48080cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f480815eae0 0x7f48080cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52964/0 (socket says 172.21.15.12:52964) 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 -- 172.21.15.12:0/893625978 learned_addr learned my addr 172.21.15.12:0/893625978 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f48077fe640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f480815e700 0x7f4808153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f4807fff640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f48080d2360 0x7f48081f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 -- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f48080d2360 msgr2=0x7f48081f79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f48080d2360 0x7f48081f79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 -- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f480815e700 msgr2=0x7f4808153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f480815e700 0x7f4808153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 -- 172.21.15.12:0/893625978 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48081f7f10 con 0x7f480815eae0 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f48077fe640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f480815e700 0x7f4808153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480cf19640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f480815eae0 0x7f48080cfc40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f480000c700 tx=0x7f480000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f4804ff9640 1 -- 172.21.15.12:0/893625978 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800018070 con 0x7f480815eae0 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f4804ff9640 1 -- 172.21.15.12:0/893625978 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480000ce10 con 0x7f480815eae0 2024-04-15T15:21:31.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48081f81e0 con 0x7f480815eae0 2024-04-15T15:21:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f4804ff9640 1 -- 172.21.15.12:0/893625978 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480000a860 con 0x7f480815eae0 2024-04-15T15:21:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.531+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48081f87a0 con 0x7f480815eae0 2024-04-15T15:21:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.532+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48080689e0 con 0x7f480815eae0 2024-04-15T15:21:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.533+0000 7f4804ff9640 1 -- 172.21.15.12:0/893625978 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f480001b070 con 0x7f480815eae0 2024-04-15T15:21:31.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.533+0000 7f4804ff9640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f47f406c920 0x7f47f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.533+0000 7f4804ff9640 1 -- 172.21.15.12:0/893625978 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f480008b320 con 0x7f480815eae0 2024-04-15T15:21:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.534+0000 7f4807fff640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f47f406c920 0x7f47f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.534+0000 7f4807fff640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f47f406c920 0x7f47f406ede0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f47f800c8e0 tx=0x7f47f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:31.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.538+0000 7f4804ff9640 1 -- 172.21.15.12:0/893625978 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f48000594a0 con 0x7f480815eae0 2024-04-15T15:21:31.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.726+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48080ba500 con 0x7f480815eae0 2024-04-15T15:21:31.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.728+0000 7f4804ff9640 1 -- 172.21.15.12:0/893625978 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f4800058da0 con 0x7f480815eae0 2024-04-15T15:21:31.729 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:31.730 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f47f406c920 msgr2=0x7f47f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f47f406c920 0x7f47f406ede0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f47f800c8e0 tx=0x7f47f8008040 comp rx=0 tx=0).stop 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f480815eae0 msgr2=0x7f48080cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f480815eae0 0x7f48080cfc40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f480000c700 tx=0x7f480000cbc0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 shutdown_connections 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f480815eae0 0x7f48080cfc40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f47f406c920 0x7f47f406ede0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f480815e700 0x7f4808153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 --2- 172.21.15.12:0/893625978 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f48080d2360 0x7f48081f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 >> 172.21.15.12:0/893625978 conn(0x7f480808bf60 msgr2=0x7f48080d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 shutdown_connections 2024-04-15T15:21:31.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.730+0000 7f480e97c640 1 -- 172.21.15.12:0/893625978 wait complete. 2024-04-15T15:21:31.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.847+0000 7f886b9ba640 1 Processor -- start 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.847+0000 7f886b9ba640 1 -- start start 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f886b9ba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f886415ebb0 0x7f886406cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f886b9ba640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f886409b780 0x7f88640b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f886b9ba640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88640cccb0 con 0x7f886409a9d0 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f886b9ba640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f88640cd8c0 con 0x7f886409b780 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f886b9ba640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f88640ced60 con 0x7f886415ebb0 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f8869f57640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f886409b780 0x7f88640b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f8869756640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f886409a9d0 0x7f886409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36490/0 (socket says 172.21.15.12:36490) 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f8869756640 1 -- 172.21.15.12:0/3032324254 learned_addr learned my addr 172.21.15.12:0/3032324254 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f8868f55640 1 --2- 172.21.15.12:0/3032324254 >> v2:172.21.15.55:3300/0 conn(0x7f886415ebb0 0x7f886406cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f8869f57640 1 -- 172.21.15.12:0/3032324254 >> v2:172.21.15.55:3300/0 conn(0x7f886415ebb0 msgr2=0x7f886406cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f8869f57640 1 --2- 172.21.15.12:0/3032324254 >> v2:172.21.15.55:3300/0 conn(0x7f886415ebb0 0x7f886406cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f8869f57640 1 -- 172.21.15.12:0/3032324254 >> v1:172.21.15.12:6789/0 conn(0x7f886409a9d0 legacy=0x7f886409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.848+0000 7f8869f57640 1 -- 172.21.15.12:0/3032324254 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88640ccf90 con 0x7f886409b780 2024-04-15T15:21:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.849+0000 7f8869f57640 1 --2- 172.21.15.12:0/3032324254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886409b780 0x7f88640b76f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88640d0560 tx=0x7f885c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00f426db36b0487 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.849+0000 7f8857fff640 1 -- 172.21.15.12:0/3032324254 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c039070 con 0x7f886409b780 2024-04-15T15:21:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.849+0000 7f8857fff640 1 -- 172.21.15.12:0/3032324254 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f885c02cc80 con 0x7f886409b780 2024-04-15T15:21:31.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.849+0000 7f8857fff640 1 -- 172.21.15.12:0/3032324254 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c034660 con 0x7f886409b780 2024-04-15T15:21:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.850+0000 7f886b9ba640 1 -- 172.21.15.12:0/3032324254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886409b780 msgr2=0x7f88640b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.850+0000 7f886b9ba640 1 --2- 172.21.15.12:0/3032324254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886409b780 0x7f88640b76f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88640d0560 tx=0x7f885c02bed0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.850+0000 7f886b9ba640 1 -- 172.21.15.12:0/3032324254 shutdown_connections 2024-04-15T15:21:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.850+0000 7f886b9ba640 1 --2- 172.21.15.12:0/3032324254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886409b780 0x7f88640b76f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.850+0000 7f886b9ba640 1 --2- 172.21.15.12:0/3032324254 >> v2:172.21.15.55:3300/0 conn(0x7f886415ebb0 0x7f886406cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.850+0000 7f886b9ba640 1 -- 172.21.15.12:0/3032324254 >> 172.21.15.12:0/3032324254 conn(0x7f88640b3fb0 msgr2=0x7f88640b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.850+0000 7f886b9ba640 1 -- 172.21.15.12:0/3032324254 shutdown_connections 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.850+0000 7f886b9ba640 1 -- 172.21.15.12:0/3032324254 wait complete. 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.851+0000 7f886b9ba640 1 Processor -- start 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.851+0000 7f886b9ba640 1 -- start start 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f886b9ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f886409a9d0 0x7f886415aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f886b9ba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f886409b780 0x7f88641611d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f886b9ba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8864161710 0x7f886415f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f886b9ba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f886406d240 con 0x7f8864161710 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f886b9ba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f88640cd550 con 0x7f886409a9d0 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f886b9ba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f88640b5310 con 0x7f886409b780 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f8869f57640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8864161710 0x7f886415f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f8869f57640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8864161710 0x7f886415f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52974/0 (socket says 172.21.15.12:52974) 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f8869f57640 1 -- 172.21.15.12:0/1977966637 learned_addr learned my addr 172.21.15.12:0/1977966637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f8869756640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f886409a9d0 0x7f886415aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.852+0000 7f8868f55640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f886409b780 0x7f88641611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8869f57640 1 -- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f886409b780 msgr2=0x7f88641611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8869f57640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f886409b780 0x7f88641611d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8869f57640 1 -- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f886409a9d0 msgr2=0x7f886415aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8869f57640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f886409a9d0 0x7f886415aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8869f57640 1 -- 172.21.15.12:0/1977966637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886415fed0 con 0x7f8864161710 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8869f57640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8864161710 0x7f886415f130 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f885c02c4e0 tx=0x7f885c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8855ffb640 1 -- 172.21.15.12:0/1977966637 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c039050 con 0x7f8864161710 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8855ffb640 1 -- 172.21.15.12:0/1977966637 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f885c045070 con 0x7f8864161710 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88641601b0 con 0x7f8864161710 2024-04-15T15:21:31.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f8855ffb640 1 -- 172.21.15.12:0/1977966637 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c03e830 con 0x7f8864161710 2024-04-15T15:21:31.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.853+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8864160770 con 0x7f8864161710 2024-04-15T15:21:31.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.854+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8864068b10 con 0x7f8864161710 2024-04-15T15:21:31.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.855+0000 7f8855ffb640 1 -- 172.21.15.12:0/1977966637 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f885c03e9d0 con 0x7f8864161710 2024-04-15T15:21:31.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.856+0000 7f8855ffb640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f884806c8f0 0x7f884806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:31.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.856+0000 7f8855ffb640 1 -- 172.21.15.12:0/1977966637 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f885c0aea00 con 0x7f8864161710 2024-04-15T15:21:31.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.856+0000 7f8869756640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f884806c8f0 0x7f884806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:31.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.857+0000 7f8869756640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f884806c8f0 0x7f884806edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8858007fd0 tx=0x7f8858007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:31.860+0000 7f8855ffb640 1 -- 172.21.15.12:0/1977966637 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8864068b10 con 0x7f8864161710 2024-04-15T15:21:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.050+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88640750a0 con 0x7f8864161710 2024-04-15T15:21:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.051+0000 7f8855ffb640 1 -- 172.21.15.12:0/1977966637 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f885c07c650 con 0x7f8864161710 2024-04-15T15:21:32.052 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:32.052 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f884806c8f0 msgr2=0x7f884806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f884806c8f0 0x7f884806edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8858007fd0 tx=0x7f8858007e80 comp rx=0 tx=0).stop 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8864161710 msgr2=0x7f886415f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8864161710 0x7f886415f130 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f885c02c4e0 tx=0x7f885c004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 shutdown_connections 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8864161710 0x7f886415f130 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f886409b780 0x7f88641611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f884806c8f0 0x7f884806edb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 --2- 172.21.15.12:0/1977966637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f886409a9d0 0x7f886415aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 >> 172.21.15.12:0/1977966637 conn(0x7f88640b3fb0 msgr2=0x7f88640d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 shutdown_connections 2024-04-15T15:21:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.053+0000 7f886b9ba640 1 -- 172.21.15.12:0/1977966637 wait complete. 2024-04-15T15:21:32.066 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-15T15:21:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.175+0000 7f96da6b1640 1 Processor -- start 2024-04-15T15:21:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.176+0000 7f96da6b1640 1 -- start start 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.176+0000 7f96da6b1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f96d40705a0 0x7f96d40af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.176+0000 7f96da6b1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96d415ebc0 con 0x7f96d40b0850 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.176+0000 7f96da6b1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96d4070a10 con 0x7f96d406b9e0 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.176+0000 7f96da6b1640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f96d40ae150 con 0x7f96d40705a0 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.176+0000 7f96cbfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f96d40b0850 0x7f96d40aeed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36498/0 (socket says 172.21.15.12:36498) 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.176+0000 7f96cbfff640 1 -- 172.21.15.12:0/3656916405 learned_addr learned my addr 172.21.15.12:0/3656916405 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d8c4e640 1 --2- 172.21.15.12:0/3656916405 >> v2:172.21.15.55:3300/0 conn(0x7f96d40705a0 0x7f96d40af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1232688590 0 0) 0x7f96d415ebc0 con 0x7f96d40b0850 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96c0003520 con 0x7f96d40b0850 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3101587941 0 0) 0x7f96c0003520 con 0x7f96d40b0850 2024-04-15T15:21:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 >> v2:172.21.15.55:3300/0 conn(0x7f96d40705a0 msgr2=0x7f96d40af5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 --2- 172.21.15.12:0/3656916405 >> v2:172.21.15.55:3300/0 conn(0x7f96d40705a0 0x7f96d40af5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 >> v1:172.21.15.55:6789/0 conn(0x7f96d406b9e0 legacy=0x7f96d406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d4070c80 con 0x7f96d40b0850 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f96bc001670 con 0x7f96d40b0850 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 >> v1:172.21.15.12:6789/0 conn(0x7f96d40b0850 legacy=0x7f96d40aeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96c00039b0 0x7f96c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96c00043b0 0x7f96c0004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c0005080 0x7f96c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f96d40ae150 con 0x7f96c00039b0 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f96c000bfe0 con 0x7f96c0005080 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f96c000c1e0 con 0x7f96c00043b0 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d3fff640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96c00039b0 0x7f96c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96d8c4e640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c0005080 0x7f96c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.177+0000 7f96cbfff640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96c00043b0 0x7f96c0004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d3fff640 1 -- 172.21.15.12:0/3656916405 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96c00043b0 msgr2=0x7f96c0004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d3fff640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96c00043b0 0x7f96c0004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d3fff640 1 -- 172.21.15.12:0/3656916405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c0005080 msgr2=0x7f96c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d3fff640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c0005080 0x7f96c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d8c4e640 1 -- 172.21.15.12:0/3656916405 reap_dead start 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d3fff640 1 -- 172.21.15.12:0/3656916405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c0003520 con 0x7f96c00039b0 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d3fff640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96c00039b0 0x7f96c0003d90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f96c4001860 tx=0x7f96c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb36dc8910db360 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c4038070 con 0x7f96c00039b0 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.178+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c4033410 con 0x7f96c00039b0 2024-04-15T15:21:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.179+0000 7f96d37fe640 1 -- 172.21.15.12:0/3656916405 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c4001430 con 0x7f96c00039b0 2024-04-15T15:21:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.179+0000 7f96da6b1640 1 -- 172.21.15.12:0/3656916405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96c00039b0 msgr2=0x7f96c0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.179+0000 7f96da6b1640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96c00039b0 0x7f96c0003d90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f96c4001860 tx=0x7f96c0005830 comp rx=0 tx=0).stop 2024-04-15T15:21:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.179+0000 7f96da6b1640 1 -- 172.21.15.12:0/3656916405 shutdown_connections 2024-04-15T15:21:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.179+0000 7f96da6b1640 1 --2- 172.21.15.12:0/3656916405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96c00039b0 0x7f96c0003d90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.179+0000 7f96da6b1640 1 -- 172.21.15.12:0/3656916405 >> 172.21.15.12:0/3656916405 conn(0x7f96d40b2e70 msgr2=0x7f96d40b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.179+0000 7f96da6b1640 1 -- 172.21.15.12:0/3656916405 shutdown_connections 2024-04-15T15:21:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.179+0000 7f96da6b1640 1 -- 172.21.15.12:0/3656916405 wait complete. 2024-04-15T15:21:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.180+0000 7f96da6b1640 1 Processor -- start 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.180+0000 7f96da6b1640 1 -- start start 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96da6b1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c00039b0 0x7f96d40b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96da6b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4148580 0x7f96d41e89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96da6b1640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96d41e8f10 0x7f96d41e9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96da6b1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f96d40ae9b0 con 0x7f96d4148580 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96da6b1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f96d40ae6d0 con 0x7f96c00039b0 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96da6b1640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f96d40cf0a0 con 0x7f96d41e8f10 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96cbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4148580 0x7f96d41e89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96cbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4148580 0x7f96d41e89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46504/0 (socket says 172.21.15.12:46504) 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96cbfff640 1 -- 172.21.15.12:0/172550410 learned_addr learned my addr 172.21.15.12:0/172550410 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.181+0000 7f96d3fff640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c00039b0 0x7f96d40b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96d8c4e640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96d41e8f10 0x7f96d41e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96cbfff640 1 -- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96d41e8f10 msgr2=0x7f96d41e9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96cbfff640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96d41e8f10 0x7f96d41e9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96cbfff640 1 -- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c00039b0 msgr2=0x7f96d40b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96cbfff640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c00039b0 0x7f96d40b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96cbfff640 1 -- 172.21.15.12:0/172550410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d40c7cf0 con 0x7f96d4148580 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96cbfff640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4148580 0x7f96d41e89d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f96bc007290 tx=0x7f96bc007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96d17fa640 1 -- 172.21.15.12:0/172550410 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00f960 con 0x7f96d4148580 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96d17fa640 1 -- 172.21.15.12:0/172550410 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96bc018070 con 0x7f96d4148580 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96d17fa640 1 -- 172.21.15.12:0/172550410 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00fd00 con 0x7f96d4148580 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d40c7fd0 con 0x7f96d4148580 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.182+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d40c5450 con 0x7f96d4148580 2024-04-15T15:21:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.183+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d40690f0 con 0x7f96d4148580 2024-04-15T15:21:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.184+0000 7f96d17fa640 1 -- 172.21.15.12:0/172550410 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f96bc01d460 con 0x7f96d4148580 2024-04-15T15:21:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.185+0000 7f96d17fa640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f96b406c920 0x7f96b406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.185+0000 7f96d17fa640 1 -- 172.21.15.12:0/172550410 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f96bc08bec0 con 0x7f96d4148580 2024-04-15T15:21:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.185+0000 7f96d3fff640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f96b406c920 0x7f96b406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.186+0000 7f96d3fff640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f96b406c920 0x7f96b406ede0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f96c4035000 tx=0x7f96c4009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.189+0000 7f96d17fa640 1 -- 172.21.15.12:0/172550410 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f96bc059fa0 con 0x7f96d4148580 2024-04-15T15:21:32.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.413+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f96d406fc30 con 0x7f96d4148580 2024-04-15T15:21:32.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.746+0000 7f96d17fa640 1 -- 172.21.15.12:0/172550410 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f96bc0598a0 con 0x7f96d4148580 2024-04-15T15:21:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f96b406c920 msgr2=0x7f96b406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f96b406c920 0x7f96b406ede0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f96c4035000 tx=0x7f96c4009030 comp rx=0 tx=0).stop 2024-04-15T15:21:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4148580 msgr2=0x7f96d41e89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4148580 0x7f96d41e89d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f96bc007290 tx=0x7f96bc007750 comp rx=0 tx=0).stop 2024-04-15T15:21:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 shutdown_connections 2024-04-15T15:21:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f96d41e8f10 0x7f96d41e9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4148580 0x7f96d41e89d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f96b406c920 0x7f96b406ede0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 --2- 172.21.15.12:0/172550410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96c00039b0 0x7f96d40b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 >> 172.21.15.12:0/172550410 conn(0x7f96d40b2e70 msgr2=0x7f96d406eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:32.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 shutdown_connections 2024-04-15T15:21:32.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.748+0000 7f96da6b1640 1 -- 172.21.15.12:0/172550410 wait complete. 2024-04-15T15:21:32.760 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-15T15:21:32.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-15T15:21:32.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.869+0000 7fae96d31640 1 Processor -- start 2024-04-15T15:21:32.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.869+0000 7fae96d31640 1 -- start start 2024-04-15T15:21:32.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae96d31640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fae90072d80 0x7fae90071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae96d31640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fae900af3e0 0x7fae900af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae96d31640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9015db40 con 0x7fae900d0690 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae96d31640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fae9015cfc0 con 0x7fae90072d80 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae96d31640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fae9015d2a0 con 0x7fae900af3e0 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae94acd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fae90072d80 0x7fae90071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae94acd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fae90072d80 0x7fae90071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46506/0 (socket says 172.21.15.12:46506) 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae952ce640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fae900d0690 0x7fae9006e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56218/0 (socket says 172.21.15.12:56218) 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae94acd640 1 -- 172.21.15.12:0/460418699 learned_addr learned my addr 172.21.15.12:0/460418699 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae8ffff640 1 --2- 172.21.15.12:0/460418699 >> v2:172.21.15.55:3300/0 conn(0x7fae900af3e0 0x7fae900af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae94acd640 1 -- 172.21.15.12:0/460418699 >> v2:172.21.15.55:3300/0 conn(0x7fae900af3e0 msgr2=0x7fae900af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae94acd640 1 --2- 172.21.15.12:0/460418699 >> v2:172.21.15.55:3300/0 conn(0x7fae900af3e0 0x7fae900af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae94acd640 1 -- 172.21.15.12:0/460418699 >> v1:172.21.15.12:6789/0 conn(0x7fae900d0690 legacy=0x7fae9006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae8f7fe640 1 -- 172.21.15.12:0/460418699 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1985098330 0 0) 0x7fae9015db40 con 0x7fae900d0690 2024-04-15T15:21:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae94acd640 1 -- 172.21.15.12:0/460418699 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9015d580 con 0x7fae90072d80 2024-04-15T15:21:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.870+0000 7fae8ffff640 1 --2- 172.21.15.12:0/460418699 >> v2:172.21.15.55:3300/0 conn(0x7fae900af3e0 0x7fae900af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.871+0000 7fae94acd640 1 --2- 172.21.15.12:0/460418699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae90072d80 0x7fae90071d10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fae900d1240 tx=0x7fae8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e58505b76a66dba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.871+0000 7fae8f7fe640 1 -- 172.21.15.12:0/460418699 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae84035020 con 0x7fae90072d80 2024-04-15T15:21:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.871+0000 7fae8f7fe640 1 -- 172.21.15.12:0/460418699 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae84031870 con 0x7fae90072d80 2024-04-15T15:21:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.871+0000 7fae8f7fe640 1 -- 172.21.15.12:0/460418699 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae84031c40 con 0x7fae90072d80 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.872+0000 7fae96d31640 1 -- 172.21.15.12:0/460418699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae90072d80 msgr2=0x7fae90071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.872+0000 7fae96d31640 1 --2- 172.21.15.12:0/460418699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae90072d80 0x7fae90071d10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fae900d1240 tx=0x7fae8402b910 comp rx=0 tx=0).stop 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.872+0000 7fae96d31640 1 -- 172.21.15.12:0/460418699 shutdown_connections 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.872+0000 7fae96d31640 1 --2- 172.21.15.12:0/460418699 >> v2:172.21.15.55:3300/0 conn(0x7fae900af3e0 0x7fae900af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.872+0000 7fae96d31640 1 --2- 172.21.15.12:0/460418699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae90072d80 0x7fae90071d10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.872+0000 7fae96d31640 1 -- 172.21.15.12:0/460418699 >> 172.21.15.12:0/460418699 conn(0x7fae9006be00 msgr2=0x7fae9006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.872+0000 7fae96d31640 1 -- 172.21.15.12:0/460418699 shutdown_connections 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.872+0000 7fae96d31640 1 -- 172.21.15.12:0/460418699 wait complete. 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.873+0000 7fae96d31640 1 Processor -- start 2024-04-15T15:21:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.873+0000 7fae96d31640 1 -- start start 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae96d31640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae90072d80 0x7fae901e51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae96d31640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae900af3e0 0x7fae900cf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae96d31640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae900d0690 0x7fae900cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae96d31640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fae9015cd10 con 0x7fae900d0690 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae96d31640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fae9015cb90 con 0x7fae90072d80 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae96d31640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fae900d0b00 con 0x7fae900af3e0 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae952ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae900d0690 0x7fae900cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae952ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae900d0690 0x7fae900cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46520/0 (socket says 172.21.15.12:46520) 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae952ce640 1 -- 172.21.15.12:0/1244775668 learned_addr learned my addr 172.21.15.12:0/1244775668 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae94acd640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae90072d80 0x7fae901e51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.874+0000 7fae8ffff640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae900af3e0 0x7fae900cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae952ce640 1 -- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae900af3e0 msgr2=0x7fae900cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae952ce640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae900af3e0 0x7fae900cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae952ce640 1 -- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae90072d80 msgr2=0x7fae901e51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae952ce640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae90072d80 0x7fae901e51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae952ce640 1 -- 172.21.15.12:0/1244775668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9014adb0 con 0x7fae900d0690 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae94acd640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae90072d80 0x7fae901e51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae8ffff640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae900af3e0 0x7fae900cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae952ce640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae900d0690 0x7fae900cd690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fae8800c700 tx=0x7fae8800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae8d7fa640 1 -- 172.21.15.12:0/1244775668 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae88018070 con 0x7fae900d0690 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae8d7fa640 1 -- 172.21.15.12:0/1244775668 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae8800ce10 con 0x7fae900d0690 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9014b090 con 0x7fae900d0690 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae8d7fa640 1 -- 172.21.15.12:0/1244775668 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8800a860 con 0x7fae900d0690 2024-04-15T15:21:32.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.875+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae9014b650 con 0x7fae900d0690 2024-04-15T15:21:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.876+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae900689e0 con 0x7fae900d0690 2024-04-15T15:21:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.877+0000 7fae8d7fa640 1 -- 172.21.15.12:0/1244775668 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae8801b070 con 0x7fae900d0690 2024-04-15T15:21:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.877+0000 7fae8d7fa640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae7406c8f0 0x7fae7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.877+0000 7fae8d7fa640 1 -- 172.21.15.12:0/1244775668 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fae8808b3d0 con 0x7fae900d0690 2024-04-15T15:21:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.877+0000 7fae94acd640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae7406c8f0 0x7fae7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:32.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.878+0000 7fae94acd640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae7406c8f0 0x7fae7406edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fae84036000 tx=0x7fae84008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:32.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:32.886+0000 7fae8d7fa640 1 -- 172.21.15.12:0/1244775668 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae880594c0 con 0x7fae900d0690 2024-04-15T15:21:33.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.112+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fae900b2f40 con 0x7fae900d0690 2024-04-15T15:21:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.748+0000 7fae8d7fa640 1 -- 172.21.15.12:0/1244775668 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7fae88058dc0 con 0x7fae900d0690 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae7406c8f0 msgr2=0x7fae7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae7406c8f0 0x7fae7406edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fae84036000 tx=0x7fae84008030 comp rx=0 tx=0).stop 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae900d0690 msgr2=0x7fae900cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae900d0690 0x7fae900cd690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fae8800c700 tx=0x7fae8800cbc0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 shutdown_connections 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae900d0690 0x7fae900cd690 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fae900af3e0 0x7fae900cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae7406c8f0 0x7fae7406edb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 --2- 172.21.15.12:0/1244775668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae90072d80 0x7fae901e51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 >> 172.21.15.12:0/1244775668 conn(0x7fae9006be00 msgr2=0x7fae900b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 shutdown_connections 2024-04-15T15:21:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.751+0000 7fae96d31640 1 -- 172.21.15.12:0/1244775668 wait complete. 2024-04-15T15:21:33.765 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-15T15:21:33.765 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:21:33.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.871+0000 7f583eaa5640 1 Processor -- start 2024-04-15T15:21:33.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.871+0000 7f583eaa5640 1 -- start start 2024-04-15T15:21:33.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583eaa5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f583815e790 0x7f58380d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583eaa5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58380d1610 0x7f58380d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583eaa5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58380d2480 con 0x7f58380cfd20 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583eaa5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f58380d2db0 con 0x7f58380d1610 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583eaa5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f58380d2760 con 0x7f583815e790 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583d042640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58380d1610 0x7f58380d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583d042640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58380d1610 0x7f58380d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46534/0 (socket says 172.21.15.12:46534) 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f5837fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f58380cfd20 0x7f58380d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56220/0 (socket says 172.21.15.12:56220) 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583d042640 1 -- 172.21.15.12:0/2157820860 learned_addr learned my addr 172.21.15.12:0/2157820860 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583c841640 1 --2- 172.21.15.12:0/2157820860 >> v2:172.21.15.55:3300/0 conn(0x7f583815e790 0x7f58380d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583d042640 1 -- 172.21.15.12:0/2157820860 >> v2:172.21.15.55:3300/0 conn(0x7f583815e790 msgr2=0x7f58380d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583d042640 1 --2- 172.21.15.12:0/2157820860 >> v2:172.21.15.55:3300/0 conn(0x7f583815e790 0x7f58380d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583d042640 1 -- 172.21.15.12:0/2157820860 >> v1:172.21.15.12:6789/0 conn(0x7f58380cfd20 legacy=0x7f58380d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:33.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f58377fe640 1 -- 172.21.15.12:0/2157820860 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1811878428 0 0) 0x7f58380d2480 con 0x7f58380cfd20 2024-04-15T15:21:33.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.872+0000 7f583d042640 1 -- 172.21.15.12:0/2157820860 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58380d2a40 con 0x7f58380d1610 2024-04-15T15:21:33.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.873+0000 7f583d042640 1 --2- 172.21.15.12:0/2157820860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380d0160 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5830009e50 tx=0x7f583002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd64377e476139a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:33.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.873+0000 7f58377fe640 1 -- 172.21.15.12:0/2157820860 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583002cb20 con 0x7f58380d1610 2024-04-15T15:21:33.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.873+0000 7f58377fe640 1 -- 172.21.15.12:0/2157820860 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f583002ccc0 con 0x7f58380d1610 2024-04-15T15:21:33.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.873+0000 7f58377fe640 1 -- 172.21.15.12:0/2157820860 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830034610 con 0x7f58380d1610 2024-04-15T15:21:33.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.874+0000 7f583eaa5640 1 -- 172.21.15.12:0/2157820860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 msgr2=0x7f58380d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.874+0000 7f583eaa5640 1 --2- 172.21.15.12:0/2157820860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380d0160 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5830009e50 tx=0x7f583002bf80 comp rx=0 tx=0).stop 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.874+0000 7f583eaa5640 1 -- 172.21.15.12:0/2157820860 shutdown_connections 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.874+0000 7f583eaa5640 1 --2- 172.21.15.12:0/2157820860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380d0160 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.874+0000 7f583eaa5640 1 --2- 172.21.15.12:0/2157820860 >> v2:172.21.15.55:3300/0 conn(0x7f583815e790 0x7f58380d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.874+0000 7f583eaa5640 1 -- 172.21.15.12:0/2157820860 >> 172.21.15.12:0/2157820860 conn(0x7f583808bf60 msgr2=0x7f58380d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.874+0000 7f583eaa5640 1 -- 172.21.15.12:0/2157820860 shutdown_connections 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.874+0000 7f583eaa5640 1 -- 172.21.15.12:0/2157820860 wait complete. 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.875+0000 7f583eaa5640 1 Processor -- start 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.875+0000 7f583eaa5640 1 -- start start 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583eaa5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58380cfd20 0x7f583815c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583eaa5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583eaa5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f583815e790 0x7f583809f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583eaa5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f583815ecf0 con 0x7f58380d1610 2024-04-15T15:21:33.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583eaa5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f583815eb70 con 0x7f583815e790 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583eaa5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f58380b74f0 con 0x7f58380cfd20 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f5837fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f5837fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46546/0 (socket says 172.21.15.12:46546) 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f5837fff640 1 -- 172.21.15.12:0/3919981989 learned_addr learned my addr 172.21.15.12:0/3919981989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583c841640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58380cfd20 0x7f583815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583d042640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f583815e790 0x7f583809f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f5837fff640 1 -- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58380cfd20 msgr2=0x7f583815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f5837fff640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58380cfd20 0x7f583815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f5837fff640 1 -- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f583815e790 msgr2=0x7f583809f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f5837fff640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f583815e790 0x7f583809f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f5837fff640 1 -- 172.21.15.12:0/3919981989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583809fcb0 con 0x7f58380d1610 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.876+0000 7f583d042640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f583815e790 0x7f583809f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.877+0000 7f5837fff640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380a0d50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f582400c4d0 tx=0x7f582400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.877+0000 7f58357fa640 1 -- 172.21.15.12:0/3919981989 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5824018070 con 0x7f58380d1610 2024-04-15T15:21:33.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.877+0000 7f58357fa640 1 -- 172.21.15.12:0/3919981989 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5824007500 con 0x7f58380d1610 2024-04-15T15:21:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.877+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583809ff90 con 0x7f58380d1610 2024-04-15T15:21:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.877+0000 7f58357fa640 1 -- 172.21.15.12:0/3919981989 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5824004a20 con 0x7f58380d1610 2024-04-15T15:21:33.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.877+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58380a0830 con 0x7f58380d1610 2024-04-15T15:21:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.878+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58380a0a30 con 0x7f58380d1610 2024-04-15T15:21:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.878+0000 7f58357fa640 1 -- 172.21.15.12:0/3919981989 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f582401b070 con 0x7f58380d1610 2024-04-15T15:21:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.879+0000 7f58357fa640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f581c06c8f0 0x7f581c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.879+0000 7f58357fa640 1 -- 172.21.15.12:0/3919981989 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f582408b180 con 0x7f58380d1610 2024-04-15T15:21:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.879+0000 7f583c841640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f581c06c8f0 0x7f581c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:33.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.880+0000 7f583c841640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f581c06c8f0 0x7f581c06edb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f582c006fd0 tx=0x7f582c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:33.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:33.883+0000 7f58357fa640 1 -- 172.21.15.12:0/3919981989 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58380a0a30 con 0x7f58380d1610 2024-04-15T15:21:34.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.109+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58380689e0 con 0x7f58380d1610 2024-04-15T15:21:34.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.110+0000 7f58357fa640 1 -- 172.21.15.12:0/3919981989 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f5824059300 con 0x7f58380d1610 2024-04-15T15:21:34.111 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:34.111 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:33.113220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:21:34.112 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-15T15:21:34.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f581c06c8f0 msgr2=0x7f581c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f581c06c8f0 0x7f581c06edb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f582c006fd0 tx=0x7f582c008040 comp rx=0 tx=0).stop 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 msgr2=0x7f58380a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380a0d50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f582400c4d0 tx=0x7f582400c990 comp rx=0 tx=0).stop 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 shutdown_connections 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f581c06c8f0 0x7f581c06edb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f583815e790 0x7f583809f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58380d1610 0x7f58380a0d50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 --2- 172.21.15.12:0/3919981989 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58380cfd20 0x7f583815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 >> 172.21.15.12:0/3919981989 conn(0x7f583808bf60 msgr2=0x7f58380b8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 shutdown_connections 2024-04-15T15:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.112+0000 7f583eaa5640 1 -- 172.21.15.12:0/3919981989 wait complete. 2024-04-15T15:21:34.124 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:34.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.227+0000 7f1213e9a640 1 Processor -- start 2024-04-15T15:21:34.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.228+0000 7f1213e9a640 1 -- start start 2024-04-15T15:21:34.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.228+0000 7f1213e9a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f120c15ecb0 0x7f120c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.228+0000 7f1213e9a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b4350 0x7f120c0b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.228+0000 7f1213e9a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f120c0b1b60 0x7f120c0ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.228+0000 7f1213e9a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f120c15f090 con 0x7f120c0b1b60 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.228+0000 7f1213e9a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f120c0cdf80 con 0x7f120c15ecb0 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.228+0000 7f1213e9a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f120c0b10d0 con 0x7f120c0b4350 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.228+0000 7f1212437640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f120c0b1b60 0x7f120c0ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1212437640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f120c0b1b60 0x7f120c0ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46548/0 (socket says 172.21.15.12:46548) 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1212437640 1 -- 172.21.15.12:0/4100402974 learned_addr learned my addr 172.21.15.12:0/4100402974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1211c36640 1 --2- 172.21.15.12:0/4100402974 >> v2:172.21.15.55:3300/0 conn(0x7f120c15ecb0 0x7f120c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1211435640 1 --2- 172.21.15.12:0/4100402974 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b4350 0x7f120c0b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1212437640 1 -- 172.21.15.12:0/4100402974 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b4350 msgr2=0x7f120c0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1212437640 1 --2- 172.21.15.12:0/4100402974 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b4350 0x7f120c0b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1212437640 1 -- 172.21.15.12:0/4100402974 >> v2:172.21.15.55:3300/0 conn(0x7f120c15ecb0 msgr2=0x7f120c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1212437640 1 --2- 172.21.15.12:0/4100402974 >> v2:172.21.15.55:3300/0 conn(0x7f120c15ecb0 0x7f120c0b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1212437640 1 -- 172.21.15.12:0/4100402974 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120c0af920 con 0x7f120c0b1b60 2024-04-15T15:21:34.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.229+0000 7f1211c36640 1 --2- 172.21.15.12:0/4100402974 >> v2:172.21.15.55:3300/0 conn(0x7f120c15ecb0 0x7f120c0b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1212437640 1 --2- 172.21.15.12:0/4100402974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b1b60 0x7f120c0ce410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f120c0d2390 tx=0x7f120402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880691cf2e8569e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1210c34640 1 -- 172.21.15.12:0/4100402974 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120402cb00 con 0x7f120c0b1b60 2024-04-15T15:21:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1210c34640 1 -- 172.21.15.12:0/4100402974 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f120402cca0 con 0x7f120c0b1b60 2024-04-15T15:21:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1210c34640 1 -- 172.21.15.12:0/4100402974 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1204034610 con 0x7f120c0b1b60 2024-04-15T15:21:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 -- 172.21.15.12:0/4100402974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b1b60 msgr2=0x7f120c0ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 --2- 172.21.15.12:0/4100402974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b1b60 0x7f120c0ce410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f120c0d2390 tx=0x7f120402bf60 comp rx=0 tx=0).stop 2024-04-15T15:21:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 -- 172.21.15.12:0/4100402974 shutdown_connections 2024-04-15T15:21:34.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 --2- 172.21.15.12:0/4100402974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b1b60 0x7f120c0ce410 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 --2- 172.21.15.12:0/4100402974 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b4350 0x7f120c0b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 --2- 172.21.15.12:0/4100402974 >> v2:172.21.15.55:3300/0 conn(0x7f120c15ecb0 0x7f120c0b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 -- 172.21.15.12:0/4100402974 >> 172.21.15.12:0/4100402974 conn(0x7f120c0b6870 msgr2=0x7f120c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 -- 172.21.15.12:0/4100402974 shutdown_connections 2024-04-15T15:21:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.230+0000 7f1213e9a640 1 -- 172.21.15.12:0/4100402974 wait complete. 2024-04-15T15:21:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.231+0000 7f1213e9a640 1 Processor -- start 2024-04-15T15:21:34.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1213e9a640 1 -- start start 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1213e9a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b1b60 0x7f120c160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1213e9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b4350 0x7f120c149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1213e9a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f120c15ecb0 0x7f120c1f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1213e9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f120c0b4790 con 0x7f120c0b4350 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1213e9a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f120c0afee0 con 0x7f120c15ecb0 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1213e9a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f120c0b4910 con 0x7f120c0b1b60 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1211435640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b4350 0x7f120c149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1211435640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b4350 0x7f120c149a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46558/0 (socket says 172.21.15.12:46558) 2024-04-15T15:21:34.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1211435640 1 -- 172.21.15.12:0/20412266 learned_addr learned my addr 172.21.15.12:0/20412266 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1212437640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f120c15ecb0 0x7f120c1f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.232+0000 7f1211c36640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b1b60 0x7f120c160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f1211435640 1 -- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b1b60 msgr2=0x7f120c160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f1211435640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b1b60 0x7f120c160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f1211435640 1 -- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f120c15ecb0 msgr2=0x7f120c1f7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f1211435640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f120c15ecb0 0x7f120c1f7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f1211435640 1 -- 172.21.15.12:0/20412266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120c160920 con 0x7f120c0b4350 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f1211435640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b4350 0x7f120c149a40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00d800 tx=0x7f11fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f12027fc640 1 -- 172.21.15.12:0/20412266 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc018020 con 0x7f120c0b4350 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f12027fc640 1 -- 172.21.15.12:0/20412266 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11fc0096a0 con 0x7f120c0b4350 2024-04-15T15:21:34.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120c160c00 con 0x7f120c0b4350 2024-04-15T15:21:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f12027fc640 1 -- 172.21.15.12:0/20412266 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc009a70 con 0x7f120c0b4350 2024-04-15T15:21:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.233+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f120c1611c0 con 0x7f120c0b4350 2024-04-15T15:21:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.234+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f120c0689e0 con 0x7f120c0b4350 2024-04-15T15:21:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.234+0000 7f12027fc640 1 -- 172.21.15.12:0/20412266 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f11fc01b020 con 0x7f120c0b4350 2024-04-15T15:21:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.235+0000 7f12027fc640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11f006c8f0 0x7f11f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.235+0000 7f12027fc640 1 -- 172.21.15.12:0/20412266 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f11fc08beb0 con 0x7f120c0b4350 2024-04-15T15:21:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.235+0000 7f1211c36640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11f006c8f0 0x7f11f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.236+0000 7f1211c36640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11f006c8f0 0x7f11f006edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1208006fd0 tx=0x7f1208008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.240+0000 7f12027fc640 1 -- 172.21.15.12:0/20412266 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11fc022030 con 0x7f120c0b4350 2024-04-15T15:21:34.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.429+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f120c0d2170 con 0x7f120c0b4350 2024-04-15T15:21:34.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.430+0000 7f12027fc640 1 -- 172.21.15.12:0/20412266 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f11fc0599e0 con 0x7f120c0b4350 2024-04-15T15:21:34.432 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:34.432 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11f006c8f0 msgr2=0x7f11f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11f006c8f0 0x7f11f006edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1208006fd0 tx=0x7f1208008040 comp rx=0 tx=0).stop 2024-04-15T15:21:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b4350 msgr2=0x7f120c149a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b4350 0x7f120c149a40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00d800 tx=0x7f11fc00dcc0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 shutdown_connections 2024-04-15T15:21:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11f006c8f0 0x7f11f006edb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f120c15ecb0 0x7f120c1f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120c0b4350 0x7f120c149a40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.432+0000 7f1213e9a640 1 --2- 172.21.15.12:0/20412266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f120c0b1b60 0x7f120c160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.433+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 >> 172.21.15.12:0/20412266 conn(0x7f120c0b6870 msgr2=0x7f120c074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.433+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 shutdown_connections 2024-04-15T15:21:34.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.433+0000 7f1213e9a640 1 -- 172.21.15.12:0/20412266 wait complete. 2024-04-15T15:21:34.445 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-15T15:21:34.445 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:21:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.548+0000 7fc122ceb640 1 Processor -- start 2024-04-15T15:21:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.548+0000 7fc122ceb640 1 -- start start 2024-04-15T15:21:34.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc122ceb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc11c15ecb0 0x7fc11c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc122ceb640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c0b1d30 0x7fc11c0af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc122ceb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc11c0b10d0 con 0x7fc11c0b4350 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc122ceb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc11c15f090 con 0x7fc11c15ecb0 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc122ceb640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc11c0cdf80 con 0x7fc11c0b1d30 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc120a87640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc11c15ecb0 0x7fc11c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc120a87640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc11c15ecb0 0x7fc11c0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46570/0 (socket says 172.21.15.12:46570) 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc113fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc11c0b4350 0x7fc11c0b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56236/0 (socket says 172.21.15.12:56236) 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc120a87640 1 -- 172.21.15.12:0/343816684 learned_addr learned my addr 172.21.15.12:0/343816684 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc121288640 1 --2- 172.21.15.12:0/343816684 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c0b1d30 0x7fc11c0af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc120a87640 1 -- 172.21.15.12:0/343816684 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c0b1d30 msgr2=0x7fc11c0af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc120a87640 1 --2- 172.21.15.12:0/343816684 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c0b1d30 0x7fc11c0af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc120a87640 1 -- 172.21.15.12:0/343816684 >> v1:172.21.15.12:6789/0 conn(0x7fc11c0b4350 legacy=0x7fc11c0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc120a87640 1 -- 172.21.15.12:0/343816684 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11c0af920 con 0x7fc11c15ecb0 2024-04-15T15:21:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.549+0000 7fc1137fe640 1 -- 172.21.15.12:0/343816684 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588931636 0 0) 0x7fc11c0b10d0 con 0x7fc11c0b4350 2024-04-15T15:21:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.550+0000 7fc120a87640 1 --2- 172.21.15.12:0/343816684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c15ecb0 0x7fc11c0b0b90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc118009870 tx=0x7fc11802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda03ad84e71a260 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.550+0000 7fc1137fe640 1 -- 172.21.15.12:0/343816684 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11802f840 con 0x7fc11c15ecb0 2024-04-15T15:21:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.550+0000 7fc1137fe640 1 -- 172.21.15.12:0/343816684 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc11802f9e0 con 0x7fc11c15ecb0 2024-04-15T15:21:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.550+0000 7fc1137fe640 1 -- 172.21.15.12:0/343816684 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11802fdb0 con 0x7fc11c15ecb0 2024-04-15T15:21:34.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.551+0000 7fc122ceb640 1 -- 172.21.15.12:0/343816684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c15ecb0 msgr2=0x7fc11c0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.551+0000 7fc122ceb640 1 --2- 172.21.15.12:0/343816684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c15ecb0 0x7fc11c0b0b90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc118009870 tx=0x7fc11802ba20 comp rx=0 tx=0).stop 2024-04-15T15:21:34.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.551+0000 7fc122ceb640 1 -- 172.21.15.12:0/343816684 shutdown_connections 2024-04-15T15:21:34.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.551+0000 7fc122ceb640 1 --2- 172.21.15.12:0/343816684 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c0b1d30 0x7fc11c0af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.551+0000 7fc122ceb640 1 --2- 172.21.15.12:0/343816684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c15ecb0 0x7fc11c0b0b90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.551+0000 7fc122ceb640 1 -- 172.21.15.12:0/343816684 >> 172.21.15.12:0/343816684 conn(0x7fc11c0b6870 msgr2=0x7fc11c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.551+0000 7fc122ceb640 1 -- 172.21.15.12:0/343816684 shutdown_connections 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.551+0000 7fc122ceb640 1 -- 172.21.15.12:0/343816684 wait complete. 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.552+0000 7fc122ceb640 1 Processor -- start 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.552+0000 7fc122ceb640 1 -- start start 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc122ceb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c0b1d30 0x7fc11c161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc122ceb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc11c0b4350 0x7fc11c148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc122ceb640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c15ecb0 0x7fc11c149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc122ceb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc11c0b47c0 con 0x7fc11c0b1d30 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc122ceb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc11c0afee0 con 0x7fc11c0b4350 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc122ceb640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc11c0d03b0 con 0x7fc11c15ecb0 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c0b1d30 0x7fc11c161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c0b1d30 0x7fc11c161ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46572/0 (socket says 172.21.15.12:46572) 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 -- 172.21.15.12:0/3294760911 learned_addr learned my addr 172.21.15.12:0/3294760911 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:34.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc121288640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c15ecb0 0x7fc11c149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 -- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c15ecb0 msgr2=0x7fc11c149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc113fff640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc11c0b4350 0x7fc11c148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c15ecb0 0x7fc11c149480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 -- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc11c0b4350 msgr2=0x7fc11c148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc11c0b4350 0x7fc11c148f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 -- 172.21.15.12:0/3294760911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11c0d0690 con 0x7fc11c0b1d30 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.553+0000 7fc120a87640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c0b1d30 0x7fc11c161ca0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc118004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.554+0000 7fc1117fa640 1 -- 172.21.15.12:0/3294760911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118037c90 con 0x7fc11c0b1d30 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.554+0000 7fc1117fa640 1 -- 172.21.15.12:0/3294760911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc118046070 con 0x7fc11c0b1d30 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.554+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc11c0d0970 con 0x7fc11c0b1d30 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.554+0000 7fc1117fa640 1 -- 172.21.15.12:0/3294760911 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118041640 con 0x7fc11c0b1d30 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.554+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc11c155900 con 0x7fc11c0b1d30 2024-04-15T15:21:34.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.555+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc11c0689e0 con 0x7fc11c0b1d30 2024-04-15T15:21:34.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.555+0000 7fc1117fa640 1 -- 172.21.15.12:0/3294760911 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc118040070 con 0x7fc11c0b1d30 2024-04-15T15:21:34.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.556+0000 7fc1117fa640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc10806c920 0x7fc10806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.556+0000 7fc1117fa640 1 -- 172.21.15.12:0/3294760911 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc1180ae590 con 0x7fc11c0b1d30 2024-04-15T15:21:34.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.556+0000 7fc113fff640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc10806c920 0x7fc10806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.556+0000 7fc113fff640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc10806c920 0x7fc10806ede0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc10c007fd0 tx=0x7fc10c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.560+0000 7fc1117fa640 1 -- 172.21.15.12:0/3294760911 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc118033030 con 0x7fc11c0b1d30 2024-04-15T15:21:34.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.744+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc11c06c7c0 con 0x7fc10806c920 2024-04-15T15:21:34.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.753+0000 7fc1117fa640 1 -- 172.21.15.12:0/3294760911 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fc11c06c7c0 con 0x7fc10806c920 2024-04-15T15:21:34.759 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:34.762 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc10806c920 msgr2=0x7fc10806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc10806c920 0x7fc10806ede0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc10c007fd0 tx=0x7fc10c009040 comp rx=0 tx=0).stop 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c0b1d30 msgr2=0x7fc11c161ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c0b1d30 0x7fc11c161ca0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc118004730 comp rx=0 tx=0).stop 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 shutdown_connections 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc11c15ecb0 0x7fc11c149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc10806c920 0x7fc10806ede0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc11c0b4350 0x7fc11c148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 --2- 172.21.15.12:0/3294760911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc11c0b1d30 0x7fc11c161ca0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 >> 172.21.15.12:0/3294760911 conn(0x7fc11c0b6870 msgr2=0x7fc11c0b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 shutdown_connections 2024-04-15T15:21:34.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.757+0000 7fc122ceb640 1 -- 172.21.15.12:0/3294760911 wait complete. 2024-04-15T15:21:34.770 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-15T15:21:34.593177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236140,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738580,"statfs":{"total":773094113280,"available":772852305920,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731482+0000","last_change":"2024-04-15T15:21:10.650644+0000","last_active":"2024-04-15T15:21:16.731482+0000","last_peered":"2024-04-15T15:21:16.731482+0000","last_clean":"2024-04-15T15:21:16.731482+0000","last_became_active":"2024-04-15T15:21:10.650315+0000","last_became_peered":"2024-04-15T15:21:10.650315+0000","last_unstale":"2024-04-15T15:21:16.731482+0000","last_undegraded":"2024-04-15T15:21:16.731482+0000","last_fullsized":"2024-04-15T15:21:16.731482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:27.659449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732827+0000","last_change":"2024-04-15T15:21:11.667138+0000","last_active":"2024-04-15T15:21:16.732827+0000","last_peered":"2024-04-15T15:21:16.732827+0000","last_clean":"2024-04-15T15:21:16.732827+0000","last_became_active":"2024-04-15T15:21:11.666799+0000","last_became_peered":"2024-04-15T15:21:11.666799+0000","last_unstale":"2024-04-15T15:21:16.732827+0000","last_undegraded":"2024-04-15T15:21:16.732827+0000","last_fullsized":"2024-04-15T15:21:16.732827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:24.468426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732274+0000","last_change":"2024-04-15T15:21:15.901408+0000","last_active":"2024-04-15T15:21:16.732274+0000","last_peered":"2024-04-15T15:21:16.732274+0000","last_clean":"2024-04-15T15:21:16.732274+0000","last_became_active":"2024-04-15T15:21:15.901032+0000","last_became_peered":"2024-04-15T15:21:15.901032+0000","last_unstale":"2024-04-15T15:21:16.732274+0000","last_undegraded":"2024-04-15T15:21:16.732274+0000","last_fullsized":"2024-04-15T15:21:16.732274+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:34.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733027+0000","last_change":"2024-04-15T15:21:10.636248+0000","last_active":"2024-04-15T15:21:16.733027+0000","last_peered":"2024-04-15T15:21:16.733027+0000","last_clean":"2024-04-15T15:21:16.733027+0000","last_became_active":"2024-04-15T15:21:10.635963+0000","last_became_peered":"2024-04-15T15:21:10.635963+0000","last_unstale":"2024-04-15T15:21:16.733027+0000","last_undegraded":"2024-04-15T15:21:16.733027+0000","last_fullsized":"2024-04-15T15:21:16.733027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:27.659449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731287+0000","last_change":"2024-04-15T15:21:11.665760+0000","last_active":"2024-04-15T15:21:16.731287+0000","last_peered":"2024-04-15T15:21:16.731287+0000","last_clean":"2024-04-15T15:21:16.731287+0000","last_became_active":"2024-04-15T15:21:11.665444+0000","last_became_peered":"2024-04-15T15:21:11.665444+0000","last_unstale":"2024-04-15T15:21:16.731287+0000","last_undegraded":"2024-04-15T15:21:16.731287+0000","last_fullsized":"2024-04-15T15:21:16.731287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.684953+0000","last_change":"2024-04-15T15:21:15.699988+0000","last_active":"2024-04-15T15:21:16.684953+0000","last_peered":"2024-04-15T15:21:16.684953+0000","last_clean":"2024-04-15T15:21:16.684953+0000","last_became_active":"2024-04-15T15:21:15.699333+0000","last_became_peered":"2024-04-15T15:21:15.699333+0000","last_unstale":"2024-04-15T15:21:16.684953+0000","last_undegraded":"2024-04-15T15:21:16.684953+0000","last_fullsized":"2024-04-15T15:21:16.684953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:26.928706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730928+0000","last_change":"2024-04-15T15:21:10.649466+0000","last_active":"2024-04-15T15:21:16.730928+0000","last_peered":"2024-04-15T15:21:16.730928+0000","last_clean":"2024-04-15T15:21:16.730928+0000","last_became_active":"2024-04-15T15:21:10.649182+0000","last_became_peered":"2024-04-15T15:21:10.649182+0000","last_unstale":"2024-04-15T15:21:16.730928+0000","last_undegraded":"2024-04-15T15:21:16.730928+0000","last_fullsized":"2024-04-15T15:21:16.730928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:01:16.245697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691301+0000","last_change":"2024-04-15T15:21:11.667011+0000","last_active":"2024-04-15T15:21:16.691301+0000","last_peered":"2024-04-15T15:21:16.691301+0000","last_clean":"2024-04-15T15:21:16.691301+0000","last_became_active":"2024-04-15T15:21:11.663992+0000","last_became_peered":"2024-04-15T15:21:11.663992+0000","last_unstale":"2024-04-15T15:21:16.691301+0000","last_undegraded":"2024-04-15T15:21:16.691301+0000","last_fullsized":"2024-04-15T15:21:16.691301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685332+0000","last_change":"2024-04-15T15:21:15.707372+0000","last_active":"2024-04-15T15:21:16.685332+0000","last_peered":"2024-04-15T15:21:16.685332+0000","last_clean":"2024-04-15T15:21:16.685332+0000","last_became_active":"2024-04-15T15:21:15.706978+0000","last_became_peered":"2024-04-15T15:21:15.706978+0000","last_unstale":"2024-04-15T15:21:16.685332+0000","last_undegraded":"2024-04-15T15:21:16.685332+0000","last_fullsized":"2024-04-15T15:21:16.685332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:32.327246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730471+0000","last_change":"2024-04-15T15:21:10.652340+0000","last_active":"2024-04-15T15:21:16.730471+0000","last_peered":"2024-04-15T15:21:16.730471+0000","last_clean":"2024-04-15T15:21:16.730471+0000","last_became_active":"2024-04-15T15:21:10.652016+0000","last_became_peered":"2024-04-15T15:21:10.652016+0000","last_unstale":"2024-04-15T15:21:16.730471+0000","last_undegraded":"2024-04-15T15:21:16.730471+0000","last_fullsized":"2024-04-15T15:21:16.730471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:54:08.208968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691646+0000","last_change":"2024-04-15T15:21:11.663781+0000","last_active":"2024-04-15T15:21:16.691646+0000","last_peered":"2024-04-15T15:21:16.691646+0000","last_clean":"2024-04-15T15:21:16.691646+0000","last_became_active":"2024-04-15T15:21:11.663232+0000","last_became_peered":"2024-04-15T15:21:11.663232+0000","last_unstale":"2024-04-15T15:21:16.691646+0000","last_undegraded":"2024-04-15T15:21:16.691646+0000","last_fullsized":"2024-04-15T15:21:16.691646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.568336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685080+0000","last_change":"2024-04-15T15:21:15.695846+0000","last_active":"2024-04-15T15:21:16.685080+0000","last_peered":"2024-04-15T15:21:16.685080+0000","last_clean":"2024-04-15T15:21:16.685080+0000","last_became_active":"2024-04-15T15:21:15.695426+0000","last_became_peered":"2024-04-15T15:21:15.695426+0000","last_unstale":"2024-04-15T15:21:16.685080+0000","last_undegraded":"2024-04-15T15:21:16.685080+0000","last_fullsized":"2024-04-15T15:21:16.685080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:28.518948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729851+0000","last_change":"2024-04-15T15:21:11.662485+0000","last_active":"2024-04-15T15:21:16.729851+0000","last_peered":"2024-04-15T15:21:16.729851+0000","last_clean":"2024-04-15T15:21:16.729851+0000","last_became_active":"2024-04-15T15:21:11.662183+0000","last_became_peered":"2024-04-15T15:21:11.662183+0000","last_unstale":"2024-04-15T15:21:16.729851+0000","last_undegraded":"2024-04-15T15:21:16.729851+0000","last_fullsized":"2024-04-15T15:21:16.729851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730547+0000","last_change":"2024-04-15T15:21:10.648295+0000","last_active":"2024-04-15T15:21:16.730547+0000","last_peered":"2024-04-15T15:21:16.730547+0000","last_clean":"2024-04-15T15:21:16.730547+0000","last_became_active":"2024-04-15T15:21:10.647985+0000","last_became_peered":"2024-04-15T15:21:10.647985+0000","last_unstale":"2024-04-15T15:21:16.730547+0000","last_undegraded":"2024-04-15T15:21:16.730547+0000","last_fullsized":"2024-04-15T15:21:16.730547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:52.730839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685664+0000","last_change":"2024-04-15T15:21:15.699611+0000","last_active":"2024-04-15T15:21:16.685664+0000","last_peered":"2024-04-15T15:21:16.685664+0000","last_clean":"2024-04-15T15:21:16.685664+0000","last_became_active":"2024-04-15T15:21:15.699129+0000","last_became_peered":"2024-04-15T15:21:15.699129+0000","last_unstale":"2024-04-15T15:21:16.685664+0000","last_undegraded":"2024-04-15T15:21:16.685664+0000","last_fullsized":"2024-04-15T15:21:16.685664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732647+0000","last_change":"2024-04-15T15:21:11.644620+0000","last_active":"2024-04-15T15:21:16.732647+0000","last_peered":"2024-04-15T15:21:16.732647+0000","last_clean":"2024-04-15T15:21:16.732647+0000","last_became_active":"2024-04-15T15:21:11.644273+0000","last_became_peered":"2024-04-15T15:21:11.644273+0000","last_unstale":"2024-04-15T15:21:16.732647+0000","last_undegraded":"2024-04-15T15:21:16.732647+0000","last_fullsized":"2024-04-15T15:21:16.732647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730443+0000","last_change":"2024-04-15T15:21:10.636457+0000","last_active":"2024-04-15T15:21:16.730443+0000","last_peered":"2024-04-15T15:21:16.730443+0000","last_clean":"2024-04-15T15:21:16.730443+0000","last_became_active":"2024-04-15T15:21:10.636143+0000","last_became_peered":"2024-04-15T15:21:10.636143+0000","last_unstale":"2024-04-15T15:21:16.730443+0000","last_undegraded":"2024-04-15T15:21:16.730443+0000","last_fullsized":"2024-04-15T15:21:16.730443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685684+0000","last_change":"2024-04-15T15:21:15.703570+0000","last_active":"2024-04-15T15:21:16.685684+0000","last_peered":"2024-04-15T15:21:16.685684+0000","last_clean":"2024-04-15T15:21:16.685684+0000","last_became_active":"2024-04-15T15:21:15.703066+0000","last_became_peered":"2024-04-15T15:21:15.703066+0000","last_unstale":"2024-04-15T15:21:16.685684+0000","last_undegraded":"2024-04-15T15:21:16.685684+0000","last_fullsized":"2024-04-15T15:21:16.685684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732393+0000","last_change":"2024-04-15T15:21:11.646745+0000","last_active":"2024-04-15T15:21:16.732393+0000","last_peered":"2024-04-15T15:21:16.732393+0000","last_clean":"2024-04-15T15:21:16.732393+0000","last_became_active":"2024-04-15T15:21:11.646375+0000","last_became_peered":"2024-04-15T15:21:11.646375+0000","last_unstale":"2024-04-15T15:21:16.732393+0000","last_undegraded":"2024-04-15T15:21:16.732393+0000","last_fullsized":"2024-04-15T15:21:16.732393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729475+0000","last_change":"2024-04-15T15:21:10.642462+0000","last_active":"2024-04-15T15:21:16.729475+0000","last_peered":"2024-04-15T15:21:16.729475+0000","last_clean":"2024-04-15T15:21:16.729475+0000","last_became_active":"2024-04-15T15:21:10.642168+0000","last_became_peered":"2024-04-15T15:21:10.642168+0000","last_unstale":"2024-04-15T15:21:16.729475+0000","last_undegraded":"2024-04-15T15:21:16.729475+0000","last_fullsized":"2024-04-15T15:21:16.729475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731042+0000","last_change":"2024-04-15T15:21:15.901600+0000","last_active":"2024-04-15T15:21:16.731042+0000","last_peered":"2024-04-15T15:21:16.731042+0000","last_clean":"2024-04-15T15:21:16.731042+0000","last_became_active":"2024-04-15T15:21:15.901233+0000","last_became_peered":"2024-04-15T15:21:15.901233+0000","last_unstale":"2024-04-15T15:21:16.731042+0000","last_undegraded":"2024-04-15T15:21:16.731042+0000","last_fullsized":"2024-04-15T15:21:16.731042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:30.138777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685208+0000","last_change":"2024-04-15T15:21:11.648506+0000","last_active":"2024-04-15T15:21:16.685208+0000","last_peered":"2024-04-15T15:21:16.685208+0000","last_clean":"2024-04-15T15:21:16.685208+0000","last_became_active":"2024-04-15T15:21:11.647972+0000","last_became_peered":"2024-04-15T15:21:11.647972+0000","last_unstale":"2024-04-15T15:21:16.685208+0000","last_undegraded":"2024-04-15T15:21:16.685208+0000","last_fullsized":"2024-04-15T15:21:16.685208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:09.853761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729694+0000","last_change":"2024-04-15T15:21:10.646955+0000","last_active":"2024-04-15T15:21:16.729694+0000","last_peered":"2024-04-15T15:21:16.729694+0000","last_clean":"2024-04-15T15:21:16.729694+0000","last_became_active":"2024-04-15T15:21:10.646308+0000","last_became_peered":"2024-04-15T15:21:10.646308+0000","last_unstale":"2024-04-15T15:21:16.729694+0000","last_undegraded":"2024-04-15T15:21:16.729694+0000","last_fullsized":"2024-04-15T15:21:16.729694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685568+0000","last_change":"2024-04-15T15:21:15.699814+0000","last_active":"2024-04-15T15:21:16.685568+0000","last_peered":"2024-04-15T15:21:16.685568+0000","last_clean":"2024-04-15T15:21:16.685568+0000","last_became_active":"2024-04-15T15:21:15.699341+0000","last_became_peered":"2024-04-15T15:21:15.699341+0000","last_unstale":"2024-04-15T15:21:16.685568+0000","last_undegraded":"2024-04-15T15:21:16.685568+0000","last_fullsized":"2024-04-15T15:21:16.685568+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:51.618858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731527+0000","last_change":"2024-04-15T15:21:11.646950+0000","last_active":"2024-04-15T15:21:16.731527+0000","last_peered":"2024-04-15T15:21:16.731527+0000","last_clean":"2024-04-15T15:21:16.731527+0000","last_became_active":"2024-04-15T15:21:11.646633+0000","last_became_peered":"2024-04-15T15:21:11.646633+0000","last_unstale":"2024-04-15T15:21:16.731527+0000","last_undegraded":"2024-04-15T15:21:16.731527+0000","last_fullsized":"2024-04-15T15:21:16.731527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:56.796018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730414+0000","last_change":"2024-04-15T15:21:10.646693+0000","last_active":"2024-04-15T15:21:16.730414+0000","last_peered":"2024-04-15T15:21:16.730414+0000","last_clean":"2024-04-15T15:21:16.730414+0000","last_became_active":"2024-04-15T15:21:10.646328+0000","last_became_peered":"2024-04-15T15:21:10.646328+0000","last_unstale":"2024-04-15T15:21:16.730414+0000","last_undegraded":"2024-04-15T15:21:16.730414+0000","last_fullsized":"2024-04-15T15:21:16.730414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685127+0000","last_change":"2024-04-15T15:21:15.902524+0000","last_active":"2024-04-15T15:21:16.685127+0000","last_peered":"2024-04-15T15:21:16.685127+0000","last_clean":"2024-04-15T15:21:16.685127+0000","last_became_active":"2024-04-15T15:21:15.901936+0000","last_became_peered":"2024-04-15T15:21:15.901936+0000","last_unstale":"2024-04-15T15:21:16.685127+0000","last_undegraded":"2024-04-15T15:21:16.685127+0000","last_fullsized":"2024-04-15T15:21:16.685127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:59.443259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733036+0000","last_change":"2024-04-15T15:21:11.657367+0000","last_active":"2024-04-15T15:21:16.733036+0000","last_peered":"2024-04-15T15:21:16.733036+0000","last_clean":"2024-04-15T15:21:16.733036+0000","last_became_active":"2024-04-15T15:21:11.657002+0000","last_became_peered":"2024-04-15T15:21:11.657002+0000","last_unstale":"2024-04-15T15:21:16.733036+0000","last_undegraded":"2024-04-15T15:21:16.733036+0000","last_fullsized":"2024-04-15T15:21:16.733036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:02.359534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731010+0000","last_change":"2024-04-15T15:21:10.644925+0000","last_active":"2024-04-15T15:21:16.731010+0000","last_peered":"2024-04-15T15:21:16.731010+0000","last_clean":"2024-04-15T15:21:16.731010+0000","last_became_active":"2024-04-15T15:21:10.644639+0000","last_became_peered":"2024-04-15T15:21:10.644639+0000","last_unstale":"2024-04-15T15:21:16.731010+0000","last_undegraded":"2024-04-15T15:21:16.731010+0000","last_fullsized":"2024-04-15T15:21:16.731010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732150+0000","last_change":"2024-04-15T15:21:15.902755+0000","last_active":"2024-04-15T15:21:16.732150+0000","last_peered":"2024-04-15T15:21:16.732150+0000","last_clean":"2024-04-15T15:21:16.732150+0000","last_became_active":"2024-04-15T15:21:15.902338+0000","last_became_peered":"2024-04-15T15:21:15.902338+0000","last_unstale":"2024-04-15T15:21:16.732150+0000","last_undegraded":"2024-04-15T15:21:16.732150+0000","last_fullsized":"2024-04-15T15:21:16.732150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:13.820849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730647+0000","last_change":"2024-04-15T15:21:11.654427+0000","last_active":"2024-04-15T15:21:16.730647+0000","last_peered":"2024-04-15T15:21:16.730647+0000","last_clean":"2024-04-15T15:21:16.730647+0000","last_became_active":"2024-04-15T15:21:11.654037+0000","last_became_peered":"2024-04-15T15:21:11.654037+0000","last_unstale":"2024-04-15T15:21:16.730647+0000","last_undegraded":"2024-04-15T15:21:16.730647+0000","last_fullsized":"2024-04-15T15:21:16.730647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731058+0000","last_change":"2024-04-15T15:21:10.649235+0000","last_active":"2024-04-15T15:21:16.731058+0000","last_peered":"2024-04-15T15:21:16.731058+0000","last_clean":"2024-04-15T15:21:16.731058+0000","last_became_active":"2024-04-15T15:21:10.648935+0000","last_became_peered":"2024-04-15T15:21:10.648935+0000","last_unstale":"2024-04-15T15:21:16.731058+0000","last_undegraded":"2024-04-15T15:21:16.731058+0000","last_fullsized":"2024-04-15T15:21:16.731058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:55.786608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729854+0000","last_change":"2024-04-15T15:21:15.899286+0000","last_active":"2024-04-15T15:21:16.729854+0000","last_peered":"2024-04-15T15:21:16.729854+0000","last_clean":"2024-04-15T15:21:16.729854+0000","last_became_active":"2024-04-15T15:21:15.898902+0000","last_became_peered":"2024-04-15T15:21:15.898902+0000","last_unstale":"2024-04-15T15:21:16.729854+0000","last_undegraded":"2024-04-15T15:21:16.729854+0000","last_fullsized":"2024-04-15T15:21:16.729854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691740+0000","last_change":"2024-04-15T15:21:11.661691+0000","last_active":"2024-04-15T15:21:16.691740+0000","last_peered":"2024-04-15T15:21:16.691740+0000","last_clean":"2024-04-15T15:21:16.691740+0000","last_became_active":"2024-04-15T15:21:11.661289+0000","last_became_peered":"2024-04-15T15:21:11.661289+0000","last_unstale":"2024-04-15T15:21:16.691740+0000","last_undegraded":"2024-04-15T15:21:16.691740+0000","last_fullsized":"2024-04-15T15:21:16.691740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:08.032185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730825+0000","last_change":"2024-04-15T15:21:10.637826+0000","last_active":"2024-04-15T15:21:16.730825+0000","last_peered":"2024-04-15T15:21:16.730825+0000","last_clean":"2024-04-15T15:21:16.730825+0000","last_became_active":"2024-04-15T15:21:10.637566+0000","last_became_peered":"2024-04-15T15:21:10.637566+0000","last_unstale":"2024-04-15T15:21:16.730825+0000","last_undegraded":"2024-04-15T15:21:16.730825+0000","last_fullsized":"2024-04-15T15:21:16.730825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:31.743027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729641+0000","last_change":"2024-04-15T15:21:15.707987+0000","last_active":"2024-04-15T15:21:16.729641+0000","last_peered":"2024-04-15T15:21:16.729641+0000","last_clean":"2024-04-15T15:21:16.729641+0000","last_became_active":"2024-04-15T15:21:15.707472+0000","last_became_peered":"2024-04-15T15:21:15.707472+0000","last_unstale":"2024-04-15T15:21:16.729641+0000","last_undegraded":"2024-04-15T15:21:16.729641+0000","last_fullsized":"2024-04-15T15:21:16.729641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685621+0000","last_change":"2024-04-15T15:21:11.644409+0000","last_active":"2024-04-15T15:21:16.685621+0000","last_peered":"2024-04-15T15:21:16.685621+0000","last_clean":"2024-04-15T15:21:16.685621+0000","last_became_active":"2024-04-15T15:21:11.644095+0000","last_became_peered":"2024-04-15T15:21:11.644095+0000","last_unstale":"2024-04-15T15:21:16.685621+0000","last_undegraded":"2024-04-15T15:21:16.685621+0000","last_fullsized":"2024-04-15T15:21:16.685621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:32.752437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691377+0000","last_change":"2024-04-15T15:21:10.646053+0000","last_active":"2024-04-15T15:21:16.691377+0000","last_peered":"2024-04-15T15:21:16.691377+0000","last_clean":"2024-04-15T15:21:16.691377+0000","last_became_active":"2024-04-15T15:21:10.645737+0000","last_became_peered":"2024-04-15T15:21:10.645737+0000","last_unstale":"2024-04-15T15:21:16.691377+0000","last_undegraded":"2024-04-15T15:21:16.691377+0000","last_fullsized":"2024-04-15T15:21:16.691377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730648+0000","last_change":"2024-04-15T15:21:15.699348+0000","last_active":"2024-04-15T15:21:16.730648+0000","last_peered":"2024-04-15T15:21:16.730648+0000","last_clean":"2024-04-15T15:21:16.730648+0000","last_became_active":"2024-04-15T15:21:15.698962+0000","last_became_peered":"2024-04-15T15:21:15.698962+0000","last_unstale":"2024-04-15T15:21:16.730648+0000","last_undegraded":"2024-04-15T15:21:16.730648+0000","last_fullsized":"2024-04-15T15:21:16.730648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:35.384335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729484+0000","last_change":"2024-04-15T15:21:11.653613+0000","last_active":"2024-04-15T15:21:16.729484+0000","last_peered":"2024-04-15T15:21:16.729484+0000","last_clean":"2024-04-15T15:21:16.729484+0000","last_became_active":"2024-04-15T15:21:11.653306+0000","last_became_peered":"2024-04-15T15:21:11.653306+0000","last_unstale":"2024-04-15T15:21:16.729484+0000","last_undegraded":"2024-04-15T15:21:16.729484+0000","last_fullsized":"2024-04-15T15:21:16.729484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731231+0000","last_change":"2024-04-15T15:21:10.636526+0000","last_active":"2024-04-15T15:21:16.731231+0000","last_peered":"2024-04-15T15:21:16.731231+0000","last_clean":"2024-04-15T15:21:16.731231+0000","last_became_active":"2024-04-15T15:21:10.636211+0000","last_became_peered":"2024-04-15T15:21:10.636211+0000","last_unstale":"2024-04-15T15:21:16.731231+0000","last_undegraded":"2024-04-15T15:21:16.731231+0000","last_fullsized":"2024-04-15T15:21:16.731231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731153+0000","last_change":"2024-04-15T15:21:15.706536+0000","last_active":"2024-04-15T15:21:16.731153+0000","last_peered":"2024-04-15T15:21:16.731153+0000","last_clean":"2024-04-15T15:21:16.731153+0000","last_became_active":"2024-04-15T15:21:15.706130+0000","last_became_peered":"2024-04-15T15:21:15.706130+0000","last_unstale":"2024-04-15T15:21:16.731153+0000","last_undegraded":"2024-04-15T15:21:16.731153+0000","last_fullsized":"2024-04-15T15:21:16.731153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:35.371338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729702+0000","last_change":"2024-04-15T15:21:11.649636+0000","last_active":"2024-04-15T15:21:16.729702+0000","last_peered":"2024-04-15T15:21:16.729702+0000","last_clean":"2024-04-15T15:21:16.729702+0000","last_became_active":"2024-04-15T15:21:11.649334+0000","last_became_peered":"2024-04-15T15:21:11.649334+0000","last_unstale":"2024-04-15T15:21:16.729702+0000","last_undegraded":"2024-04-15T15:21:16.729702+0000","last_fullsized":"2024-04-15T15:21:16.729702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:01:17.255107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730694+0000","last_change":"2024-04-15T15:21:10.647209+0000","last_active":"2024-04-15T15:21:16.730694+0000","last_peered":"2024-04-15T15:21:16.730694+0000","last_clean":"2024-04-15T15:21:16.730694+0000","last_became_active":"2024-04-15T15:21:10.646650+0000","last_became_peered":"2024-04-15T15:21:10.646650+0000","last_unstale":"2024-04-15T15:21:16.730694+0000","last_undegraded":"2024-04-15T15:21:16.730694+0000","last_fullsized":"2024-04-15T15:21:16.730694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:31.743027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731537+0000","last_change":"2024-04-15T15:21:15.901283+0000","last_active":"2024-04-15T15:21:16.731537+0000","last_peered":"2024-04-15T15:21:16.731537+0000","last_clean":"2024-04-15T15:21:16.731537+0000","last_became_active":"2024-04-15T15:21:15.900931+0000","last_became_peered":"2024-04-15T15:21:15.900931+0000","last_unstale":"2024-04-15T15:21:16.731537+0000","last_undegraded":"2024-04-15T15:21:16.731537+0000","last_fullsized":"2024-04-15T15:21:16.731537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:35.371338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732331+0000","last_change":"2024-04-15T15:21:11.664990+0000","last_active":"2024-04-15T15:21:16.732331+0000","last_peered":"2024-04-15T15:21:16.732331+0000","last_clean":"2024-04-15T15:21:16.732331+0000","last_became_active":"2024-04-15T15:21:11.664706+0000","last_became_peered":"2024-04-15T15:21:11.664706+0000","last_unstale":"2024-04-15T15:21:16.732331+0000","last_undegraded":"2024-04-15T15:21:16.732331+0000","last_fullsized":"2024-04-15T15:21:16.732331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691196+0000","last_change":"2024-04-15T15:21:10.645516+0000","last_active":"2024-04-15T15:21:16.691196+0000","last_peered":"2024-04-15T15:21:16.691196+0000","last_clean":"2024-04-15T15:21:16.691196+0000","last_became_active":"2024-04-15T15:21:10.645191+0000","last_became_peered":"2024-04-15T15:21:10.645191+0000","last_unstale":"2024-04-15T15:21:16.691196+0000","last_undegraded":"2024-04-15T15:21:16.691196+0000","last_fullsized":"2024-04-15T15:21:16.691196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730801+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:21:16.730801+0000","last_peered":"2024-04-15T15:21:16.730801+0000","last_clean":"2024-04-15T15:21:16.730801+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:21:16.730801+0000","last_undegraded":"2024-04-15T15:21:16.730801+0000","last_fullsized":"2024-04-15T15:21:16.730801+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730072+0000","last_change":"2024-04-15T15:21:15.901627+0000","last_active":"2024-04-15T15:21:16.730072+0000","last_peered":"2024-04-15T15:21:16.730072+0000","last_clean":"2024-04-15T15:21:16.730072+0000","last_became_active":"2024-04-15T15:21:15.901132+0000","last_became_peered":"2024-04-15T15:21:15.901132+0000","last_unstale":"2024-04-15T15:21:16.730072+0000","last_undegraded":"2024-04-15T15:21:16.730072+0000","last_fullsized":"2024-04-15T15:21:16.730072+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730781+0000","last_change":"2024-04-15T15:21:11.665709+0000","last_active":"2024-04-15T15:21:16.730781+0000","last_peered":"2024-04-15T15:21:16.730781+0000","last_clean":"2024-04-15T15:21:16.730781+0000","last_became_active":"2024-04-15T15:21:11.665254+0000","last_became_peered":"2024-04-15T15:21:11.665254+0000","last_unstale":"2024-04-15T15:21:16.730781+0000","last_undegraded":"2024-04-15T15:21:16.730781+0000","last_fullsized":"2024-04-15T15:21:16.730781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:31.333813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730984+0000","last_change":"2024-04-15T15:21:10.644131+0000","last_active":"2024-04-15T15:21:16.730984+0000","last_peered":"2024-04-15T15:21:16.730984+0000","last_clean":"2024-04-15T15:21:16.730984+0000","last_became_active":"2024-04-15T15:21:10.643844+0000","last_became_peered":"2024-04-15T15:21:10.643844+0000","last_unstale":"2024-04-15T15:21:16.730984+0000","last_undegraded":"2024-04-15T15:21:16.730984+0000","last_fullsized":"2024-04-15T15:21:16.730984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:16:25.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729900+0000","last_change":"2024-04-15T15:21:15.899284+0000","last_active":"2024-04-15T15:21:16.729900+0000","last_peered":"2024-04-15T15:21:16.729900+0000","last_clean":"2024-04-15T15:21:16.729900+0000","last_became_active":"2024-04-15T15:21:15.898822+0000","last_became_peered":"2024-04-15T15:21:15.898822+0000","last_unstale":"2024-04-15T15:21:16.729900+0000","last_undegraded":"2024-04-15T15:21:16.729900+0000","last_fullsized":"2024-04-15T15:21:16.729900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:24.117458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691307+0000","last_change":"2024-04-15T15:21:11.663221+0000","last_active":"2024-04-15T15:21:16.691307+0000","last_peered":"2024-04-15T15:21:16.691307+0000","last_clean":"2024-04-15T15:21:16.691307+0000","last_became_active":"2024-04-15T15:21:11.662827+0000","last_became_peered":"2024-04-15T15:21:11.662827+0000","last_unstale":"2024-04-15T15:21:16.691307+0000","last_undegraded":"2024-04-15T15:21:16.691307+0000","last_fullsized":"2024-04-15T15:21:16.691307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:15.476006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730374+0000","last_change":"2024-04-15T15:21:10.636999+0000","last_active":"2024-04-15T15:21:16.730374+0000","last_peered":"2024-04-15T15:21:16.730374+0000","last_clean":"2024-04-15T15:21:16.730374+0000","last_became_active":"2024-04-15T15:21:10.636729+0000","last_became_peered":"2024-04-15T15:21:10.636729+0000","last_unstale":"2024-04-15T15:21:16.730374+0000","last_undegraded":"2024-04-15T15:21:16.730374+0000","last_fullsized":"2024-04-15T15:21:16.730374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730116+0000","last_change":"2024-04-15T15:21:15.708002+0000","last_active":"2024-04-15T15:21:16.730116+0000","last_peered":"2024-04-15T15:21:16.730116+0000","last_clean":"2024-04-15T15:21:16.730116+0000","last_became_active":"2024-04-15T15:21:15.707636+0000","last_became_peered":"2024-04-15T15:21:15.707636+0000","last_unstale":"2024-04-15T15:21:16.730116+0000","last_undegraded":"2024-04-15T15:21:16.730116+0000","last_fullsized":"2024-04-15T15:21:16.730116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730705+0000","last_change":"2024-04-15T15:21:11.662157+0000","last_active":"2024-04-15T15:21:16.730705+0000","last_peered":"2024-04-15T15:21:16.730705+0000","last_clean":"2024-04-15T15:21:16.730705+0000","last_became_active":"2024-04-15T15:21:11.661885+0000","last_became_peered":"2024-04-15T15:21:11.661885+0000","last_unstale":"2024-04-15T15:21:16.730705+0000","last_undegraded":"2024-04-15T15:21:16.730705+0000","last_fullsized":"2024-04-15T15:21:16.730705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:30.561749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731734+0000","last_change":"2024-04-15T15:21:10.635657+0000","last_active":"2024-04-15T15:21:16.731734+0000","last_peered":"2024-04-15T15:21:16.731734+0000","last_clean":"2024-04-15T15:21:16.731734+0000","last_became_active":"2024-04-15T15:21:10.635339+0000","last_became_peered":"2024-04-15T15:21:10.635339+0000","last_unstale":"2024-04-15T15:21:16.731734+0000","last_undegraded":"2024-04-15T15:21:16.731734+0000","last_fullsized":"2024-04-15T15:21:16.731734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690911+0000","last_change":"2024-04-15T15:21:15.902642+0000","last_active":"2024-04-15T15:21:16.690911+0000","last_peered":"2024-04-15T15:21:16.690911+0000","last_clean":"2024-04-15T15:21:16.690911+0000","last_became_active":"2024-04-15T15:21:15.902252+0000","last_became_peered":"2024-04-15T15:21:15.902252+0000","last_unstale":"2024-04-15T15:21:16.690911+0000","last_undegraded":"2024-04-15T15:21:16.690911+0000","last_fullsized":"2024-04-15T15:21:16.690911+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:13.820849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730900+0000","last_change":"2024-04-15T15:21:11.658612+0000","last_active":"2024-04-15T15:21:16.730900+0000","last_peered":"2024-04-15T15:21:16.730900+0000","last_clean":"2024-04-15T15:21:16.730900+0000","last_became_active":"2024-04-15T15:21:11.658312+0000","last_became_peered":"2024-04-15T15:21:11.658312+0000","last_unstale":"2024-04-15T15:21:16.730900+0000","last_undegraded":"2024-04-15T15:21:16.730900+0000","last_fullsized":"2024-04-15T15:21:16.730900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:15.476006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685731+0000","last_change":"2024-04-15T15:21:10.636964+0000","last_active":"2024-04-15T15:21:16.685731+0000","last_peered":"2024-04-15T15:21:16.685731+0000","last_clean":"2024-04-15T15:21:16.685731+0000","last_became_active":"2024-04-15T15:21:10.636641+0000","last_became_peered":"2024-04-15T15:21:10.636641+0000","last_unstale":"2024-04-15T15:21:16.685731+0000","last_undegraded":"2024-04-15T15:21:16.685731+0000","last_fullsized":"2024-04-15T15:21:16.685731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:27:05.012468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.684974+0000","last_change":"2024-04-15T15:21:15.706953+0000","last_active":"2024-04-15T15:21:16.684974+0000","last_peered":"2024-04-15T15:21:16.684974+0000","last_clean":"2024-04-15T15:21:16.684974+0000","last_became_active":"2024-04-15T15:21:15.706465+0000","last_became_peered":"2024-04-15T15:21:15.706465+0000","last_unstale":"2024-04-15T15:21:16.684974+0000","last_undegraded":"2024-04-15T15:21:16.684974+0000","last_fullsized":"2024-04-15T15:21:16.684974+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:06.410056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731056+0000","last_change":"2024-04-15T15:21:11.658373+0000","last_active":"2024-04-15T15:21:16.731056+0000","last_peered":"2024-04-15T15:21:16.731056+0000","last_clean":"2024-04-15T15:21:16.731056+0000","last_became_active":"2024-04-15T15:21:11.658069+0000","last_became_peered":"2024-04-15T15:21:11.658069+0000","last_unstale":"2024-04-15T15:21:16.731056+0000","last_undegraded":"2024-04-15T15:21:16.731056+0000","last_fullsized":"2024-04-15T15:21:16.731056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:31.320816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730149+0000","last_change":"2024-04-15T15:21:10.637607+0000","last_active":"2024-04-15T15:21:16.730149+0000","last_peered":"2024-04-15T15:21:16.730149+0000","last_clean":"2024-04-15T15:21:16.730149+0000","last_became_active":"2024-04-15T15:21:10.637277+0000","last_became_peered":"2024-04-15T15:21:10.637277+0000","last_unstale":"2024-04-15T15:21:16.730149+0000","last_undegraded":"2024-04-15T15:21:16.730149+0000","last_fullsized":"2024-04-15T15:21:16.730149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:03:48.192888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731366+0000","last_change":"2024-04-15T15:21:15.696964+0000","last_active":"2024-04-15T15:21:16.731366+0000","last_peered":"2024-04-15T15:21:16.731366+0000","last_clean":"2024-04-15T15:21:16.731366+0000","last_became_active":"2024-04-15T15:21:15.696559+0000","last_became_peered":"2024-04-15T15:21:15.696559+0000","last_unstale":"2024-04-15T15:21:16.731366+0000","last_undegraded":"2024-04-15T15:21:16.731366+0000","last_fullsized":"2024-04-15T15:21:16.731366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:35.384335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729725+0000","last_change":"2024-04-15T15:21:11.661036+0000","last_active":"2024-04-15T15:21:16.729725+0000","last_peered":"2024-04-15T15:21:16.729725+0000","last_clean":"2024-04-15T15:21:16.729725+0000","last_became_active":"2024-04-15T15:21:11.660616+0000","last_became_peered":"2024-04-15T15:21:11.660616+0000","last_unstale":"2024-04-15T15:21:16.729725+0000","last_undegraded":"2024-04-15T15:21:16.729725+0000","last_fullsized":"2024-04-15T15:21:16.729725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729989+0000","last_change":"2024-04-15T15:21:10.644599+0000","last_active":"2024-04-15T15:21:16.729989+0000","last_peered":"2024-04-15T15:21:16.729989+0000","last_clean":"2024-04-15T15:21:16.729989+0000","last_became_active":"2024-04-15T15:21:10.644274+0000","last_became_peered":"2024-04-15T15:21:10.644274+0000","last_unstale":"2024-04-15T15:21:16.729989+0000","last_undegraded":"2024-04-15T15:21:16.729989+0000","last_fullsized":"2024-04-15T15:21:16.729989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732952+0000","last_change":"2024-04-15T15:21:15.901481+0000","last_active":"2024-04-15T15:21:16.732952+0000","last_peered":"2024-04-15T15:21:16.732952+0000","last_clean":"2024-04-15T15:21:16.732952+0000","last_became_active":"2024-04-15T15:21:15.901096+0000","last_became_peered":"2024-04-15T15:21:15.901096+0000","last_unstale":"2024-04-15T15:21:16.732952+0000","last_undegraded":"2024-04-15T15:21:16.732952+0000","last_fullsized":"2024-04-15T15:21:16.732952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:38.490295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730312+0000","last_change":"2024-04-15T15:21:11.658341+0000","last_active":"2024-04-15T15:21:16.730312+0000","last_peered":"2024-04-15T15:21:16.730312+0000","last_clean":"2024-04-15T15:21:16.730312+0000","last_became_active":"2024-04-15T15:21:11.658024+0000","last_became_peered":"2024-04-15T15:21:11.658024+0000","last_unstale":"2024-04-15T15:21:16.730312+0000","last_undegraded":"2024-04-15T15:21:16.730312+0000","last_fullsized":"2024-04-15T15:21:16.730312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.568336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691308+0000","last_change":"2024-04-15T15:21:10.645818+0000","last_active":"2024-04-15T15:21:16.691308+0000","last_peered":"2024-04-15T15:21:16.691308+0000","last_clean":"2024-04-15T15:21:16.691308+0000","last_became_active":"2024-04-15T15:21:10.645371+0000","last_became_peered":"2024-04-15T15:21:10.645371+0000","last_unstale":"2024-04-15T15:21:16.691308+0000","last_undegraded":"2024-04-15T15:21:16.691308+0000","last_fullsized":"2024-04-15T15:21:16.691308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733065+0000","last_change":"2024-04-15T15:21:15.899911+0000","last_active":"2024-04-15T15:21:16.733065+0000","last_peered":"2024-04-15T15:21:16.733065+0000","last_clean":"2024-04-15T15:21:16.733065+0000","last_became_active":"2024-04-15T15:21:15.899428+0000","last_became_peered":"2024-04-15T15:21:15.899428+0000","last_unstale":"2024-04-15T15:21:16.733065+0000","last_undegraded":"2024-04-15T15:21:16.733065+0000","last_fullsized":"2024-04-15T15:21:16.733065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:18:40.268133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731100+0000","last_change":"2024-04-15T15:21:11.659032+0000","last_active":"2024-04-15T15:21:16.731100+0000","last_peered":"2024-04-15T15:21:16.731100+0000","last_clean":"2024-04-15T15:21:16.731100+0000","last_became_active":"2024-04-15T15:21:11.658769+0000","last_became_peered":"2024-04-15T15:21:11.658769+0000","last_unstale":"2024-04-15T15:21:16.731100+0000","last_undegraded":"2024-04-15T15:21:16.731100+0000","last_fullsized":"2024-04-15T15:21:16.731100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730358+0000","last_change":"2024-04-15T15:21:10.640715+0000","last_active":"2024-04-15T15:21:16.730358+0000","last_peered":"2024-04-15T15:21:16.730358+0000","last_clean":"2024-04-15T15:21:16.730358+0000","last_became_active":"2024-04-15T15:21:10.640364+0000","last_became_peered":"2024-04-15T15:21:10.640364+0000","last_unstale":"2024-04-15T15:21:16.730358+0000","last_undegraded":"2024-04-15T15:21:16.730358+0000","last_fullsized":"2024-04-15T15:21:16.730358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:54:53.368896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691556+0000","last_change":"2024-04-15T15:21:15.901574+0000","last_active":"2024-04-15T15:21:16.691556+0000","last_peered":"2024-04-15T15:21:16.691556+0000","last_clean":"2024-04-15T15:21:16.691556+0000","last_became_active":"2024-04-15T15:21:15.901144+0000","last_became_peered":"2024-04-15T15:21:15.901144+0000","last_unstale":"2024-04-15T15:21:16.691556+0000","last_undegraded":"2024-04-15T15:21:16.691556+0000","last_fullsized":"2024-04-15T15:21:16.691556+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:34.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730711+0000","last_change":"2024-04-15T15:21:11.655252+0000","last_active":"2024-04-15T15:21:16.730711+0000","last_peered":"2024-04-15T15:21:16.730711+0000","last_clean":"2024-04-15T15:21:16.730711+0000","last_became_active":"2024-04-15T15:21:11.654839+0000","last_became_peered":"2024-04-15T15:21:11.654839+0000","last_unstale":"2024-04-15T15:21:16.730711+0000","last_undegraded":"2024-04-15T15:21:16.730711+0000","last_fullsized":"2024-04-15T15:21:16.730711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:08.032185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:21.251862+0000","last_change":"2024-04-15T15:21:10.645952+0000","last_active":"2024-04-15T15:21:21.251862+0000","last_peered":"2024-04-15T15:21:21.251862+0000","last_clean":"2024-04-15T15:21:21.251862+0000","last_became_active":"2024-04-15T15:21:10.645618+0000","last_became_peered":"2024-04-15T15:21:10.645618+0000","last_unstale":"2024-04-15T15:21:21.251862+0000","last_undegraded":"2024-04-15T15:21:21.251862+0000","last_fullsized":"2024-04-15T15:21:21.251862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730096+0000","last_change":"2024-04-15T15:21:15.703360+0000","last_active":"2024-04-15T15:21:16.730096+0000","last_peered":"2024-04-15T15:21:16.730096+0000","last_clean":"2024-04-15T15:21:16.730096+0000","last_became_active":"2024-04-15T15:21:15.702868+0000","last_became_peered":"2024-04-15T15:21:15.702868+0000","last_unstale":"2024-04-15T15:21:16.730096+0000","last_undegraded":"2024-04-15T15:21:16.730096+0000","last_fullsized":"2024-04-15T15:21:16.730096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685743+0000","last_change":"2024-04-15T15:21:11.659799+0000","last_active":"2024-04-15T15:21:16.685743+0000","last_peered":"2024-04-15T15:21:16.685743+0000","last_clean":"2024-04-15T15:21:16.685743+0000","last_became_active":"2024-04-15T15:21:11.659521+0000","last_became_peered":"2024-04-15T15:21:11.659521+0000","last_unstale":"2024-04-15T15:21:16.685743+0000","last_undegraded":"2024-04-15T15:21:16.685743+0000","last_fullsized":"2024-04-15T15:21:16.685743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:38.472181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730267+0000","last_change":"2024-04-15T15:21:10.645938+0000","last_active":"2024-04-15T15:21:16.730267+0000","last_peered":"2024-04-15T15:21:16.730267+0000","last_clean":"2024-04-15T15:21:16.730267+0000","last_became_active":"2024-04-15T15:21:10.645642+0000","last_became_peered":"2024-04-15T15:21:10.645642+0000","last_unstale":"2024-04-15T15:21:16.730267+0000","last_undegraded":"2024-04-15T15:21:16.730267+0000","last_fullsized":"2024-04-15T15:21:16.730267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685481+0000","last_change":"2024-04-15T15:21:15.705076+0000","last_active":"2024-04-15T15:21:16.685481+0000","last_peered":"2024-04-15T15:21:16.685481+0000","last_clean":"2024-04-15T15:21:16.685481+0000","last_became_active":"2024-04-15T15:21:15.704627+0000","last_became_peered":"2024-04-15T15:21:15.704627+0000","last_unstale":"2024-04-15T15:21:16.685481+0000","last_undegraded":"2024-04-15T15:21:16.685481+0000","last_fullsized":"2024-04-15T15:21:16.685481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:24.117458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731387+0000","last_change":"2024-04-15T15:21:11.665544+0000","last_active":"2024-04-15T15:21:16.731387+0000","last_peered":"2024-04-15T15:21:16.731387+0000","last_clean":"2024-04-15T15:21:16.731387+0000","last_became_active":"2024-04-15T15:21:11.665263+0000","last_became_peered":"2024-04-15T15:21:11.665263+0000","last_unstale":"2024-04-15T15:21:16.731387+0000","last_undegraded":"2024-04-15T15:21:16.731387+0000","last_fullsized":"2024-04-15T15:21:16.731387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691599+0000","last_change":"2024-04-15T15:21:10.648496+0000","last_active":"2024-04-15T15:21:16.691599+0000","last_peered":"2024-04-15T15:21:16.691599+0000","last_clean":"2024-04-15T15:21:16.691599+0000","last_became_active":"2024-04-15T15:21:10.648222+0000","last_became_peered":"2024-04-15T15:21:10.648222+0000","last_unstale":"2024-04-15T15:21:16.691599+0000","last_undegraded":"2024-04-15T15:21:16.691599+0000","last_fullsized":"2024-04-15T15:21:16.691599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732523+0000","last_change":"2024-04-15T15:21:15.699583+0000","last_active":"2024-04-15T15:21:16.732523+0000","last_peered":"2024-04-15T15:21:16.732523+0000","last_clean":"2024-04-15T15:21:16.732523+0000","last_became_active":"2024-04-15T15:21:15.699119+0000","last_became_peered":"2024-04-15T15:21:15.699119+0000","last_unstale":"2024-04-15T15:21:16.732523+0000","last_undegraded":"2024-04-15T15:21:16.732523+0000","last_fullsized":"2024-04-15T15:21:16.732523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:41:57.140691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691502+0000","last_change":"2024-04-15T15:21:11.663796+0000","last_active":"2024-04-15T15:21:16.691502+0000","last_peered":"2024-04-15T15:21:16.691502+0000","last_clean":"2024-04-15T15:21:16.691502+0000","last_became_active":"2024-04-15T15:21:11.663502+0000","last_became_peered":"2024-04-15T15:21:11.663502+0000","last_unstale":"2024-04-15T15:21:16.691502+0000","last_undegraded":"2024-04-15T15:21:16.691502+0000","last_fullsized":"2024-04-15T15:21:16.691502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:31.333813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730207+0000","last_change":"2024-04-15T15:21:10.651176+0000","last_active":"2024-04-15T15:21:16.730207+0000","last_peered":"2024-04-15T15:21:16.730207+0000","last_clean":"2024-04-15T15:21:16.730207+0000","last_became_active":"2024-04-15T15:21:10.650301+0000","last_became_peered":"2024-04-15T15:21:10.650301+0000","last_unstale":"2024-04-15T15:21:16.730207+0000","last_undegraded":"2024-04-15T15:21:16.730207+0000","last_fullsized":"2024-04-15T15:21:16.730207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732313+0000","last_change":"2024-04-15T15:21:15.899209+0000","last_active":"2024-04-15T15:21:16.732313+0000","last_peered":"2024-04-15T15:21:16.732313+0000","last_clean":"2024-04-15T15:21:16.732313+0000","last_became_active":"2024-04-15T15:21:15.898799+0000","last_became_peered":"2024-04-15T15:21:15.898799+0000","last_unstale":"2024-04-15T15:21:16.732313+0000","last_undegraded":"2024-04-15T15:21:16.732313+0000","last_fullsized":"2024-04-15T15:21:16.732313+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:16.350687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731404+0000","last_change":"2024-04-15T15:21:15.902104+0000","last_active":"2024-04-15T15:21:16.731404+0000","last_peered":"2024-04-15T15:21:16.731404+0000","last_clean":"2024-04-15T15:21:16.731404+0000","last_became_active":"2024-04-15T15:21:15.901711+0000","last_became_peered":"2024-04-15T15:21:15.901711+0000","last_unstale":"2024-04-15T15:21:16.731404+0000","last_undegraded":"2024-04-15T15:21:16.731404+0000","last_fullsized":"2024-04-15T15:21:16.731404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:30.138777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729279+0000","last_change":"2024-04-15T15:21:11.649924+0000","last_active":"2024-04-15T15:21:16.729279+0000","last_peered":"2024-04-15T15:21:16.729279+0000","last_clean":"2024-04-15T15:21:16.729279+0000","last_became_active":"2024-04-15T15:21:11.649654+0000","last_became_peered":"2024-04-15T15:21:11.649654+0000","last_unstale":"2024-04-15T15:21:16.729279+0000","last_undegraded":"2024-04-15T15:21:16.729279+0000","last_fullsized":"2024-04-15T15:21:16.729279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691073+0000","last_change":"2024-04-15T15:21:10.647906+0000","last_active":"2024-04-15T15:21:16.691073+0000","last_peered":"2024-04-15T15:21:16.691073+0000","last_clean":"2024-04-15T15:21:16.691073+0000","last_became_active":"2024-04-15T15:21:10.647594+0000","last_became_peered":"2024-04-15T15:21:10.647594+0000","last_unstale":"2024-04-15T15:21:16.691073+0000","last_undegraded":"2024-04-15T15:21:16.691073+0000","last_fullsized":"2024-04-15T15:21:16.691073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729962+0000","last_change":"2024-04-15T15:21:15.707054+0000","last_active":"2024-04-15T15:21:16.729962+0000","last_peered":"2024-04-15T15:21:16.729962+0000","last_clean":"2024-04-15T15:21:16.729962+0000","last_became_active":"2024-04-15T15:21:15.706636+0000","last_became_peered":"2024-04-15T15:21:15.706636+0000","last_unstale":"2024-04-15T15:21:16.729962+0000","last_undegraded":"2024-04-15T15:21:16.729962+0000","last_fullsized":"2024-04-15T15:21:16.729962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:51.618858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732275+0000","last_change":"2024-04-15T15:21:11.644995+0000","last_active":"2024-04-15T15:21:16.732275+0000","last_peered":"2024-04-15T15:21:16.732275+0000","last_clean":"2024-04-15T15:21:16.732275+0000","last_became_active":"2024-04-15T15:21:11.644654+0000","last_became_peered":"2024-04-15T15:21:11.644654+0000","last_unstale":"2024-04-15T15:21:16.732275+0000","last_undegraded":"2024-04-15T15:21:16.732275+0000","last_fullsized":"2024-04-15T15:21:16.732275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732479+0000","last_change":"2024-04-15T15:21:10.645892+0000","last_active":"2024-04-15T15:21:16.732479+0000","last_peered":"2024-04-15T15:21:16.732479+0000","last_clean":"2024-04-15T15:21:16.732479+0000","last_became_active":"2024-04-15T15:21:10.645615+0000","last_became_peered":"2024-04-15T15:21:10.645615+0000","last_unstale":"2024-04-15T15:21:16.732479+0000","last_undegraded":"2024-04-15T15:21:16.732479+0000","last_fullsized":"2024-04-15T15:21:16.732479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:16:25.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730156+0000","last_change":"2024-04-15T15:21:10.646124+0000","last_active":"2024-04-15T15:21:16.730156+0000","last_peered":"2024-04-15T15:21:16.730156+0000","last_clean":"2024-04-15T15:21:16.730156+0000","last_became_active":"2024-04-15T15:21:10.644818+0000","last_became_peered":"2024-04-15T15:21:10.644818+0000","last_unstale":"2024-04-15T15:21:16.730156+0000","last_undegraded":"2024-04-15T15:21:16.730156+0000","last_fullsized":"2024-04-15T15:21:16.730156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730262+0000","last_change":"2024-04-15T15:21:11.648824+0000","last_active":"2024-04-15T15:21:16.730262+0000","last_peered":"2024-04-15T15:21:16.730262+0000","last_clean":"2024-04-15T15:21:16.730262+0000","last_became_active":"2024-04-15T15:21:11.648490+0000","last_became_peered":"2024-04-15T15:21:11.648490+0000","last_unstale":"2024-04-15T15:21:16.730262+0000","last_undegraded":"2024-04-15T15:21:16.730262+0000","last_fullsized":"2024-04-15T15:21:16.730262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730777+0000","last_change":"2024-04-15T15:21:15.699586+0000","last_active":"2024-04-15T15:21:16.730777+0000","last_peered":"2024-04-15T15:21:16.730777+0000","last_clean":"2024-04-15T15:21:16.730777+0000","last_became_active":"2024-04-15T15:21:15.699130+0000","last_became_peered":"2024-04-15T15:21:15.699130+0000","last_unstale":"2024-04-15T15:21:16.730777+0000","last_undegraded":"2024-04-15T15:21:16.730777+0000","last_fullsized":"2024-04-15T15:21:16.730777+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:47.552269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732768+0000","last_change":"2024-04-15T15:21:10.641801+0000","last_active":"2024-04-15T15:21:16.732768+0000","last_peered":"2024-04-15T15:21:16.732768+0000","last_clean":"2024-04-15T15:21:16.732768+0000","last_became_active":"2024-04-15T15:21:10.641511+0000","last_became_peered":"2024-04-15T15:21:10.641511+0000","last_unstale":"2024-04-15T15:21:16.732768+0000","last_undegraded":"2024-04-15T15:21:16.732768+0000","last_fullsized":"2024-04-15T15:21:16.732768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:03:48.192888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685152+0000","last_change":"2024-04-15T15:21:11.650188+0000","last_active":"2024-04-15T15:21:16.685152+0000","last_peered":"2024-04-15T15:21:16.685152+0000","last_clean":"2024-04-15T15:21:16.685152+0000","last_became_active":"2024-04-15T15:21:11.649838+0000","last_became_peered":"2024-04-15T15:21:11.649838+0000","last_unstale":"2024-04-15T15:21:16.685152+0000","last_undegraded":"2024-04-15T15:21:16.685152+0000","last_fullsized":"2024-04-15T15:21:16.685152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:12.989671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732888+0000","last_change":"2024-04-15T15:21:15.699639+0000","last_active":"2024-04-15T15:21:16.732888+0000","last_peered":"2024-04-15T15:21:16.732888+0000","last_clean":"2024-04-15T15:21:16.732888+0000","last_became_active":"2024-04-15T15:21:15.699252+0000","last_became_peered":"2024-04-15T15:21:15.699252+0000","last_unstale":"2024-04-15T15:21:16.732888+0000","last_undegraded":"2024-04-15T15:21:16.732888+0000","last_fullsized":"2024-04-15T15:21:16.732888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:02.471663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729618+0000","last_change":"2024-04-15T15:21:11.662366+0000","last_active":"2024-04-15T15:21:16.729618+0000","last_peered":"2024-04-15T15:21:16.729618+0000","last_clean":"2024-04-15T15:21:16.729618+0000","last_became_active":"2024-04-15T15:21:11.662082+0000","last_became_peered":"2024-04-15T15:21:11.662082+0000","last_unstale":"2024-04-15T15:21:16.729618+0000","last_undegraded":"2024-04-15T15:21:16.729618+0000","last_fullsized":"2024-04-15T15:21:16.729618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731145+0000","last_change":"2024-04-15T15:21:11.667867+0000","last_active":"2024-04-15T15:21:16.731145+0000","last_peered":"2024-04-15T15:21:16.731145+0000","last_clean":"2024-04-15T15:21:16.731145+0000","last_became_active":"2024-04-15T15:21:11.667664+0000","last_became_peered":"2024-04-15T15:21:11.667664+0000","last_unstale":"2024-04-15T15:21:16.731145+0000","last_undegraded":"2024-04-15T15:21:16.731145+0000","last_fullsized":"2024-04-15T15:21:16.731145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:32.752437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691237+0000","last_change":"2024-04-15T15:21:11.662949+0000","last_active":"2024-04-15T15:21:16.691237+0000","last_peered":"2024-04-15T15:21:16.691237+0000","last_clean":"2024-04-15T15:21:16.691237+0000","last_became_active":"2024-04-15T15:21:11.662014+0000","last_became_peered":"2024-04-15T15:21:11.662014+0000","last_unstale":"2024-04-15T15:21:16.691237+0000","last_undegraded":"2024-04-15T15:21:16.691237+0000","last_fullsized":"2024-04-15T15:21:16.691237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:20.066936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730170+0000","last_change":"2024-04-15T15:21:11.643845+0000","last_active":"2024-04-15T15:21:16.730170+0000","last_peered":"2024-04-15T15:21:16.730170+0000","last_clean":"2024-04-15T15:21:16.730170+0000","last_became_active":"2024-04-15T15:21:11.643459+0000","last_became_peered":"2024-04-15T15:21:11.643459+0000","last_unstale":"2024-04-15T15:21:16.730170+0000","last_undegraded":"2024-04-15T15:21:16.730170+0000","last_fullsized":"2024-04-15T15:21:16.730170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:43.501747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733169+0000","last_change":"2024-04-15T15:21:11.649864+0000","last_active":"2024-04-15T15:21:16.733169+0000","last_peered":"2024-04-15T15:21:16.733169+0000","last_clean":"2024-04-15T15:21:16.733169+0000","last_became_active":"2024-04-15T15:21:11.649174+0000","last_became_peered":"2024-04-15T15:21:11.649174+0000","last_unstale":"2024-04-15T15:21:16.733169+0000","last_undegraded":"2024-04-15T15:21:16.733169+0000","last_fullsized":"2024-04-15T15:21:16.733169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:55.392737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730146+0000","last_change":"2024-04-15T15:21:11.649917+0000","last_active":"2024-04-15T15:21:16.730146+0000","last_peered":"2024-04-15T15:21:16.730146+0000","last_clean":"2024-04-15T15:21:16.730146+0000","last_became_active":"2024-04-15T15:21:11.649627+0000","last_became_peered":"2024-04-15T15:21:11.649627+0000","last_unstale":"2024-04-15T15:21:16.730146+0000","last_undegraded":"2024-04-15T15:21:16.730146+0000","last_fullsized":"2024-04-15T15:21:16.730146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730543+0000","last_change":"2024-04-15T15:21:11.666666+0000","last_active":"2024-04-15T15:21:16.730543+0000","last_peered":"2024-04-15T15:21:16.730543+0000","last_clean":"2024-04-15T15:21:16.730543+0000","last_became_active":"2024-04-15T15:21:11.666354+0000","last_became_peered":"2024-04-15T15:21:11.666354+0000","last_unstale":"2024-04-15T15:21:16.730543+0000","last_undegraded":"2024-04-15T15:21:16.730543+0000","last_fullsized":"2024-04-15T15:21:16.730543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732965+0000","last_change":"2024-04-15T15:21:11.649875+0000","last_active":"2024-04-15T15:21:16.732965+0000","last_peered":"2024-04-15T15:21:16.732965+0000","last_clean":"2024-04-15T15:21:16.732965+0000","last_became_active":"2024-04-15T15:21:11.649492+0000","last_became_peered":"2024-04-15T15:21:11.649492+0000","last_unstale":"2024-04-15T15:21:16.732965+0000","last_undegraded":"2024-04-15T15:21:16.732965+0000","last_fullsized":"2024-04-15T15:21:16.732965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690980+0000","last_change":"2024-04-15T15:21:11.663545+0000","last_active":"2024-04-15T15:21:16.690980+0000","last_peered":"2024-04-15T15:21:16.690980+0000","last_clean":"2024-04-15T15:21:16.690980+0000","last_became_active":"2024-04-15T15:21:11.662983+0000","last_became_peered":"2024-04-15T15:21:11.662983+0000","last_unstale":"2024-04-15T15:21:16.690980+0000","last_undegraded":"2024-04-15T15:21:16.690980+0000","last_fullsized":"2024-04-15T15:21:16.690980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:24.468426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732374+0000","last_change":"2024-04-15T15:21:11.649581+0000","last_active":"2024-04-15T15:21:16.732374+0000","last_peered":"2024-04-15T15:21:16.732374+0000","last_clean":"2024-04-15T15:21:16.732374+0000","last_became_active":"2024-04-15T15:21:11.649255+0000","last_became_peered":"2024-04-15T15:21:11.649255+0000","last_unstale":"2024-04-15T15:21:16.732374+0000","last_undegraded":"2024-04-15T15:21:16.732374+0000","last_fullsized":"2024-04-15T15:21:16.732374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691111+0000","last_change":"2024-04-15T15:21:11.667227+0000","last_active":"2024-04-15T15:21:16.691111+0000","last_peered":"2024-04-15T15:21:16.691111+0000","last_clean":"2024-04-15T15:21:16.691111+0000","last_became_active":"2024-04-15T15:21:11.663970+0000","last_became_peered":"2024-04-15T15:21:11.663970+0000","last_unstale":"2024-04-15T15:21:16.691111+0000","last_undegraded":"2024-04-15T15:21:16.691111+0000","last_fullsized":"2024-04-15T15:21:16.691111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:02.359534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729911+0000","last_change":"2024-04-15T15:21:11.659069+0000","last_active":"2024-04-15T15:21:16.729911+0000","last_peered":"2024-04-15T15:21:16.729911+0000","last_clean":"2024-04-15T15:21:16.729911+0000","last_became_active":"2024-04-15T15:21:11.658826+0000","last_became_peered":"2024-04-15T15:21:11.658826+0000","last_unstale":"2024-04-15T15:21:16.729911+0000","last_undegraded":"2024-04-15T15:21:16.729911+0000","last_fullsized":"2024-04-15T15:21:16.729911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:34:00.069973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730161+0000","last_change":"2024-04-15T15:21:11.662189+0000","last_active":"2024-04-15T15:21:16.730161+0000","last_peered":"2024-04-15T15:21:16.730161+0000","last_clean":"2024-04-15T15:21:16.730161+0000","last_became_active":"2024-04-15T15:21:11.661850+0000","last_became_peered":"2024-04-15T15:21:11.661850+0000","last_unstale":"2024-04-15T15:21:16.730161+0000","last_undegraded":"2024-04-15T15:21:16.730161+0000","last_fullsized":"2024-04-15T15:21:16.730161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730978+0000","last_change":"2024-04-15T15:21:11.656094+0000","last_active":"2024-04-15T15:21:16.730978+0000","last_peered":"2024-04-15T15:21:16.730978+0000","last_clean":"2024-04-15T15:21:16.730978+0000","last_became_active":"2024-04-15T15:21:11.655585+0000","last_became_peered":"2024-04-15T15:21:11.655585+0000","last_unstale":"2024-04-15T15:21:16.730978+0000","last_undegraded":"2024-04-15T15:21:16.730978+0000","last_fullsized":"2024-04-15T15:21:16.730978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:26.088255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731645+0000","last_change":"2024-04-15T15:21:11.663357+0000","last_active":"2024-04-15T15:21:16.731645+0000","last_peered":"2024-04-15T15:21:16.731645+0000","last_clean":"2024-04-15T15:21:16.731645+0000","last_became_active":"2024-04-15T15:21:11.663051+0000","last_became_peered":"2024-04-15T15:21:11.663051+0000","last_unstale":"2024-04-15T15:21:16.731645+0000","last_undegraded":"2024-04-15T15:21:16.731645+0000","last_fullsized":"2024-04-15T15:21:16.731645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:53.740250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691911+0000","last_change":"2024-04-15T15:21:11.661796+0000","last_active":"2024-04-15T15:21:16.691911+0000","last_peered":"2024-04-15T15:21:16.691911+0000","last_clean":"2024-04-15T15:21:16.691911+0000","last_became_active":"2024-04-15T15:21:11.661288+0000","last_became_peered":"2024-04-15T15:21:11.661288+0000","last_unstale":"2024-04-15T15:21:16.691911+0000","last_undegraded":"2024-04-15T15:21:16.691911+0000","last_fullsized":"2024-04-15T15:21:16.691911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:26.088255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690954+0000","last_change":"2024-04-15T15:21:11.663287+0000","last_active":"2024-04-15T15:21:16.690954+0000","last_peered":"2024-04-15T15:21:16.690954+0000","last_clean":"2024-04-15T15:21:16.690954+0000","last_became_active":"2024-04-15T15:21:11.662684+0000","last_became_peered":"2024-04-15T15:21:11.662684+0000","last_unstale":"2024-04-15T15:21:16.690954+0000","last_undegraded":"2024-04-15T15:21:16.690954+0000","last_fullsized":"2024-04-15T15:21:16.690954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729440+0000","last_change":"2024-04-15T15:21:11.657278+0000","last_active":"2024-04-15T15:21:16.729440+0000","last_peered":"2024-04-15T15:21:16.729440+0000","last_clean":"2024-04-15T15:21:16.729440+0000","last_became_active":"2024-04-15T15:21:11.656878+0000","last_became_peered":"2024-04-15T15:21:11.656878+0000","last_unstale":"2024-04-15T15:21:16.729440+0000","last_undegraded":"2024-04-15T15:21:16.729440+0000","last_fullsized":"2024-04-15T15:21:16.729440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730993+0000","last_change":"2024-04-15T15:21:11.646900+0000","last_active":"2024-04-15T15:21:16.730993+0000","last_peered":"2024-04-15T15:21:16.730993+0000","last_clean":"2024-04-15T15:21:16.730993+0000","last_became_active":"2024-04-15T15:21:11.646504+0000","last_became_peered":"2024-04-15T15:21:11.646504+0000","last_unstale":"2024-04-15T15:21:16.730993+0000","last_undegraded":"2024-04-15T15:21:16.730993+0000","last_fullsized":"2024-04-15T15:21:16.730993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732733+0000","last_change":"2024-04-15T15:21:11.666961+0000","last_active":"2024-04-15T15:21:16.732733+0000","last_peered":"2024-04-15T15:21:16.732733+0000","last_clean":"2024-04-15T15:21:16.732733+0000","last_became_active":"2024-04-15T15:21:11.666715+0000","last_became_peered":"2024-04-15T15:21:11.666715+0000","last_unstale":"2024-04-15T15:21:16.732733+0000","last_undegraded":"2024-04-15T15:21:16.732733+0000","last_fullsized":"2024-04-15T15:21:16.732733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685618+0000","last_change":"2024-04-15T15:21:11.662851+0000","last_active":"2024-04-15T15:21:16.685618+0000","last_peered":"2024-04-15T15:21:16.685618+0000","last_clean":"2024-04-15T15:21:16.685618+0000","last_became_active":"2024-04-15T15:21:11.662536+0000","last_became_peered":"2024-04-15T15:21:11.662536+0000","last_unstale":"2024-04-15T15:21:16.685618+0000","last_undegraded":"2024-04-15T15:21:16.685618+0000","last_fullsized":"2024-04-15T15:21:16.685618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:53.932651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730237+0000","last_change":"2024-04-15T15:21:11.670651+0000","last_active":"2024-04-15T15:21:16.730237+0000","last_peered":"2024-04-15T15:21:16.730237+0000","last_clean":"2024-04-15T15:21:16.730237+0000","last_became_active":"2024-04-15T15:21:11.670354+0000","last_became_peered":"2024-04-15T15:21:11.670354+0000","last_unstale":"2024-04-15T15:21:16.730237+0000","last_undegraded":"2024-04-15T15:21:16.730237+0000","last_fullsized":"2024-04-15T15:21:16.730237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730494+0000","last_change":"2024-04-15T15:21:11.670652+0000","last_active":"2024-04-15T15:21:16.730494+0000","last_peered":"2024-04-15T15:21:16.730494+0000","last_clean":"2024-04-15T15:21:16.730494+0000","last_became_active":"2024-04-15T15:21:11.670396+0000","last_became_peered":"2024-04-15T15:21:11.670396+0000","last_unstale":"2024-04-15T15:21:16.730494+0000","last_undegraded":"2024-04-15T15:21:16.730494+0000","last_fullsized":"2024-04-15T15:21:16.730494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730636+0000","last_change":"2024-04-15T15:21:11.665992+0000","last_active":"2024-04-15T15:21:16.730636+0000","last_peered":"2024-04-15T15:21:16.730636+0000","last_clean":"2024-04-15T15:21:16.730636+0000","last_became_active":"2024-04-15T15:21:11.665531+0000","last_became_peered":"2024-04-15T15:21:11.665531+0000","last_unstale":"2024-04-15T15:21:16.730636+0000","last_undegraded":"2024-04-15T15:21:16.730636+0000","last_fullsized":"2024-04-15T15:21:16.730636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:09.770327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732177+0000","last_change":"2024-04-15T15:21:11.665238+0000","last_active":"2024-04-15T15:21:16.732177+0000","last_peered":"2024-04-15T15:21:16.732177+0000","last_clean":"2024-04-15T15:21:16.732177+0000","last_became_active":"2024-04-15T15:21:11.664931+0000","last_became_peered":"2024-04-15T15:21:11.664931+0000","last_unstale":"2024-04-15T15:21:16.732177+0000","last_undegraded":"2024-04-15T15:21:16.732177+0000","last_fullsized":"2024-04-15T15:21:16.732177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731045+0000","last_change":"2024-04-15T15:21:11.655222+0000","last_active":"2024-04-15T15:21:16.731045+0000","last_peered":"2024-04-15T15:21:16.731045+0000","last_clean":"2024-04-15T15:21:16.731045+0000","last_became_active":"2024-04-15T15:21:11.654826+0000","last_became_peered":"2024-04-15T15:21:11.654826+0000","last_unstale":"2024-04-15T15:21:16.731045+0000","last_undegraded":"2024-04-15T15:21:16.731045+0000","last_fullsized":"2024-04-15T15:21:16.731045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:20.066936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690947+0000","last_change":"2024-04-15T15:21:11.662632+0000","last_active":"2024-04-15T15:21:16.690947+0000","last_peered":"2024-04-15T15:21:16.690947+0000","last_clean":"2024-04-15T15:21:16.690947+0000","last_became_active":"2024-04-15T15:21:11.661632+0000","last_became_peered":"2024-04-15T15:21:11.661632+0000","last_unstale":"2024-04-15T15:21:16.690947+0000","last_undegraded":"2024-04-15T15:21:16.690947+0000","last_fullsized":"2024-04-15T15:21:16.690947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730098+0000","last_change":"2024-04-15T15:21:11.655451+0000","last_active":"2024-04-15T15:21:16.730098+0000","last_peered":"2024-04-15T15:21:16.730098+0000","last_clean":"2024-04-15T15:21:16.730098+0000","last_became_active":"2024-04-15T15:21:11.655138+0000","last_became_peered":"2024-04-15T15:21:11.655138+0000","last_unstale":"2024-04-15T15:21:16.730098+0000","last_undegraded":"2024-04-15T15:21:16.730098+0000","last_fullsized":"2024-04-15T15:21:16.730098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691160+0000","last_change":"2024-04-15T15:21:11.662669+0000","last_active":"2024-04-15T15:21:16.691160+0000","last_peered":"2024-04-15T15:21:16.691160+0000","last_clean":"2024-04-15T15:21:16.691160+0000","last_became_active":"2024-04-15T15:21:11.662385+0000","last_became_peered":"2024-04-15T15:21:11.662385+0000","last_unstale":"2024-04-15T15:21:16.691160+0000","last_undegraded":"2024-04-15T15:21:16.691160+0000","last_fullsized":"2024-04-15T15:21:16.691160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:55.392737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731209+0000","last_change":"2024-04-15T15:21:11.640925+0000","last_active":"2024-04-15T15:21:16.731209+0000","last_peered":"2024-04-15T15:21:16.731209+0000","last_clean":"2024-04-15T15:21:16.731209+0000","last_became_active":"2024-04-15T15:21:11.640618+0000","last_became_peered":"2024-04-15T15:21:11.640618+0000","last_unstale":"2024-04-15T15:21:16.731209+0000","last_undegraded":"2024-04-15T15:21:16.731209+0000","last_fullsized":"2024-04-15T15:21:16.731209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730901+0000","last_change":"2024-04-15T15:21:11.653243+0000","last_active":"2024-04-15T15:21:16.730901+0000","last_peered":"2024-04-15T15:21:16.730901+0000","last_clean":"2024-04-15T15:21:16.730901+0000","last_became_active":"2024-04-15T15:21:11.652962+0000","last_became_peered":"2024-04-15T15:21:11.652962+0000","last_unstale":"2024-04-15T15:21:16.730901+0000","last_undegraded":"2024-04-15T15:21:16.730901+0000","last_fullsized":"2024-04-15T15:21:16.730901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730496+0000","last_change":"2024-04-15T15:21:11.662774+0000","last_active":"2024-04-15T15:21:16.730496+0000","last_peered":"2024-04-15T15:21:16.730496+0000","last_clean":"2024-04-15T15:21:16.730496+0000","last_became_active":"2024-04-15T15:21:11.662488+0000","last_became_peered":"2024-04-15T15:21:11.662488+0000","last_unstale":"2024-04-15T15:21:16.730496+0000","last_undegraded":"2024-04-15T15:21:16.730496+0000","last_fullsized":"2024-04-15T15:21:16.730496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:09.923283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:21:34.776 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:21:34.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.913+0000 7f46ea0a5640 1 Processor -- start 2024-04-15T15:21:34.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.913+0000 7f46ea0a5640 1 -- start start 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46ea0a5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f46e4071d10 0x7f46e40d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46ea0a5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e415db40 con 0x7f46e40d0bd0 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46ea0a5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e4072170 con 0x7f46e4072d80 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46ea0a5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f46e40d1040 con 0x7f46e4071d10 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46e2ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f46e4071d10 0x7f46e40d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46e3fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f46e40d0bd0 0x7f46e406e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56252/0 (socket says 172.21.15.12:56252) 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46e3fff640 1 -- 172.21.15.12:0/397780118 learned_addr learned my addr 172.21.15.12:0/397780118 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46e2ffd640 1 -- 172.21.15.12:0/397780118 >> v1:172.21.15.55:6789/0 conn(0x7f46e4072d80 legacy=0x7f46e40af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46e2ffd640 1 -- 172.21.15.12:0/397780118 >> v1:172.21.15.12:6789/0 conn(0x7f46e40d0bd0 legacy=0x7f46e406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46e2ffd640 1 -- 172.21.15.12:0/397780118 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e415cfc0 con 0x7f46e4071d10 2024-04-15T15:21:34.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.914+0000 7f46e27fc640 1 -- 172.21.15.12:0/397780118 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3139126933 0 0) 0x7f46e415db40 con 0x7f46e40d0bd0 2024-04-15T15:21:34.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.915+0000 7f46e2ffd640 1 --2- 172.21.15.12:0/397780118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e4071d10 0x7f46e40d0690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46d80097a0 tx=0x7f46d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3455f3f39caed7de server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.915+0000 7f46e27fc640 1 -- 172.21.15.12:0/397780118 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8035020 con 0x7f46e4071d10 2024-04-15T15:21:34.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.915+0000 7f46e27fc640 1 -- 172.21.15.12:0/397780118 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d8031830 con 0x7f46e4071d10 2024-04-15T15:21:34.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.915+0000 7f46e27fc640 1 -- 172.21.15.12:0/397780118 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8031c00 con 0x7f46e4071d10 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.916+0000 7f46ea0a5640 1 -- 172.21.15.12:0/397780118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e4071d10 msgr2=0x7f46e40d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.916+0000 7f46ea0a5640 1 --2- 172.21.15.12:0/397780118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e4071d10 0x7f46e40d0690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46d80097a0 tx=0x7f46d802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.916+0000 7f46ea0a5640 1 -- 172.21.15.12:0/397780118 shutdown_connections 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.916+0000 7f46ea0a5640 1 --2- 172.21.15.12:0/397780118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e4071d10 0x7f46e40d0690 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.916+0000 7f46ea0a5640 1 -- 172.21.15.12:0/397780118 >> 172.21.15.12:0/397780118 conn(0x7f46e406be00 msgr2=0x7f46e406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.916+0000 7f46ea0a5640 1 -- 172.21.15.12:0/397780118 shutdown_connections 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.916+0000 7f46ea0a5640 1 -- 172.21.15.12:0/397780118 wait complete. 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.917+0000 7f46ea0a5640 1 Processor -- start 2024-04-15T15:21:34.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.917+0000 7f46ea0a5640 1 -- start start 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46ea0a5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e4071d10 0x7f46e41f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46ea0a5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f46e4072d80 0x7f46e40b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46ea0a5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e40d0bd0 0x7f46e41e6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46ea0a5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46e415d860 con 0x7f46e40d0bd0 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46ea0a5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f46e415d580 con 0x7f46e4071d10 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46ea0a5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f46e40afaf0 con 0x7f46e4072d80 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e40d0bd0 0x7f46e41e6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e40d0bd0 0x7f46e41e6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46598/0 (socket says 172.21.15.12:46598) 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 -- 172.21.15.12:0/2055898564 learned_addr learned my addr 172.21.15.12:0/2055898564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:34.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e37fe640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e4071d10 0x7f46e41f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e2ffd640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f46e4072d80 0x7f46e40b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 -- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f46e4072d80 msgr2=0x7f46e40b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f46e4072d80 0x7f46e40b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 -- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e4071d10 msgr2=0x7f46e41f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e4071d10 0x7f46e41f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 -- 172.21.15.12:0/2055898564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e4157630 con 0x7f46e40d0bd0 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.918+0000 7f46e3fff640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e40d0bd0 0x7f46e41e6400 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00c700 tx=0x7f46dc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.919+0000 7f46cbfff640 1 -- 172.21.15.12:0/2055898564 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc018070 con 0x7f46e40d0bd0 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.919+0000 7f46cbfff640 1 -- 172.21.15.12:0/2055898564 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46dc00ce10 con 0x7f46e40d0bd0 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.919+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e4157910 con 0x7f46e40d0bd0 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.919+0000 7f46cbfff640 1 -- 172.21.15.12:0/2055898564 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc00a800 con 0x7f46e40d0bd0 2024-04-15T15:21:34.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.919+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e4157ed0 con 0x7f46e40d0bd0 2024-04-15T15:21:34.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.920+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e40689e0 con 0x7f46e40d0bd0 2024-04-15T15:21:34.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.920+0000 7f46cbfff640 1 -- 172.21.15.12:0/2055898564 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f46dc01b070 con 0x7f46e40d0bd0 2024-04-15T15:21:34.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.921+0000 7f46cbfff640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46d006c8f0 0x7f46d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:34.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.921+0000 7f46cbfff640 1 -- 172.21.15.12:0/2055898564 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f46dc08b3c0 con 0x7f46e40d0bd0 2024-04-15T15:21:34.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.921+0000 7f46e37fe640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46d006c8f0 0x7f46d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:34.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.921+0000 7f46e37fe640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46d006c8f0 0x7f46d006edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f46d4000f60 tx=0x7f46d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:34.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:34.925+0000 7f46cbfff640 1 -- 172.21.15.12:0/2055898564 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f46dc058dd0 con 0x7f46e40d0bd0 2024-04-15T15:21:35.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.108+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46e4073cb0 con 0x7f46d006c8f0 2024-04-15T15:21:35.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.117+0000 7f46cbfff640 1 -- 172.21.15.12:0/2055898564 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f46e4073cb0 con 0x7f46d006c8f0 2024-04-15T15:21:35.122 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:35.124 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:21:35.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.120+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46d006c8f0 msgr2=0x7f46d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.120+0000 7f46ea0a5640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46d006c8f0 0x7f46d006edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f46d4000f60 tx=0x7f46d4009040 comp rx=0 tx=0).stop 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.120+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e40d0bd0 msgr2=0x7f46e41e6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.120+0000 7f46ea0a5640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e40d0bd0 0x7f46e41e6400 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00c700 tx=0x7f46dc00cbc0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.120+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 shutdown_connections 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.120+0000 7f46ea0a5640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46e40d0bd0 0x7f46e41e6400 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.120+0000 7f46ea0a5640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f46e4072d80 0x7f46e40b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.120+0000 7f46ea0a5640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46d006c8f0 0x7f46d006edb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.121+0000 7f46ea0a5640 1 --2- 172.21.15.12:0/2055898564 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e4071d10 0x7f46e41f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.121+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 >> 172.21.15.12:0/2055898564 conn(0x7f46e406be00 msgr2=0x7f46e415c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.121+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 shutdown_connections 2024-04-15T15:21:35.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.121+0000 7f46ea0a5640 1 -- 172.21.15.12:0/2055898564 wait complete. 2024-04-15T15:21:35.133 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-15T15:21:34.593177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236140,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738580,"statfs":{"total":773094113280,"available":772852305920,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731482+0000","last_change":"2024-04-15T15:21:10.650644+0000","last_active":"2024-04-15T15:21:16.731482+0000","last_peered":"2024-04-15T15:21:16.731482+0000","last_clean":"2024-04-15T15:21:16.731482+0000","last_became_active":"2024-04-15T15:21:10.650315+0000","last_became_peered":"2024-04-15T15:21:10.650315+0000","last_unstale":"2024-04-15T15:21:16.731482+0000","last_undegraded":"2024-04-15T15:21:16.731482+0000","last_fullsized":"2024-04-15T15:21:16.731482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:27.659449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732827+0000","last_change":"2024-04-15T15:21:11.667138+0000","last_active":"2024-04-15T15:21:16.732827+0000","last_peered":"2024-04-15T15:21:16.732827+0000","last_clean":"2024-04-15T15:21:16.732827+0000","last_became_active":"2024-04-15T15:21:11.666799+0000","last_became_peered":"2024-04-15T15:21:11.666799+0000","last_unstale":"2024-04-15T15:21:16.732827+0000","last_undegraded":"2024-04-15T15:21:16.732827+0000","last_fullsized":"2024-04-15T15:21:16.732827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:24.468426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732274+0000","last_change":"2024-04-15T15:21:15.901408+0000","last_active":"2024-04-15T15:21:16.732274+0000","last_peered":"2024-04-15T15:21:16.732274+0000","last_clean":"2024-04-15T15:21:16.732274+0000","last_became_active":"2024-04-15T15:21:15.901032+0000","last_became_peered":"2024-04-15T15:21:15.901032+0000","last_unstale":"2024-04-15T15:21:16.732274+0000","last_undegraded":"2024-04-15T15:21:16.732274+0000","last_fullsized":"2024-04-15T15:21:16.732274+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:34.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733027+0000","last_change":"2024-04-15T15:21:10.636248+0000","last_active":"2024-04-15T15:21:16.733027+0000","last_peered":"2024-04-15T15:21:16.733027+0000","last_clean":"2024-04-15T15:21:16.733027+0000","last_became_active":"2024-04-15T15:21:10.635963+0000","last_became_peered":"2024-04-15T15:21:10.635963+0000","last_unstale":"2024-04-15T15:21:16.733027+0000","last_undegraded":"2024-04-15T15:21:16.733027+0000","last_fullsized":"2024-04-15T15:21:16.733027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:27.659449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731287+0000","last_change":"2024-04-15T15:21:11.665760+0000","last_active":"2024-04-15T15:21:16.731287+0000","last_peered":"2024-04-15T15:21:16.731287+0000","last_clean":"2024-04-15T15:21:16.731287+0000","last_became_active":"2024-04-15T15:21:11.665444+0000","last_became_peered":"2024-04-15T15:21:11.665444+0000","last_unstale":"2024-04-15T15:21:16.731287+0000","last_undegraded":"2024-04-15T15:21:16.731287+0000","last_fullsized":"2024-04-15T15:21:16.731287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.684953+0000","last_change":"2024-04-15T15:21:15.699988+0000","last_active":"2024-04-15T15:21:16.684953+0000","last_peered":"2024-04-15T15:21:16.684953+0000","last_clean":"2024-04-15T15:21:16.684953+0000","last_became_active":"2024-04-15T15:21:15.699333+0000","last_became_peered":"2024-04-15T15:21:15.699333+0000","last_unstale":"2024-04-15T15:21:16.684953+0000","last_undegraded":"2024-04-15T15:21:16.684953+0000","last_fullsized":"2024-04-15T15:21:16.684953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:26.928706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730928+0000","last_change":"2024-04-15T15:21:10.649466+0000","last_active":"2024-04-15T15:21:16.730928+0000","last_peered":"2024-04-15T15:21:16.730928+0000","last_clean":"2024-04-15T15:21:16.730928+0000","last_became_active":"2024-04-15T15:21:10.649182+0000","last_became_peered":"2024-04-15T15:21:10.649182+0000","last_unstale":"2024-04-15T15:21:16.730928+0000","last_undegraded":"2024-04-15T15:21:16.730928+0000","last_fullsized":"2024-04-15T15:21:16.730928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:01:16.245697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691301+0000","last_change":"2024-04-15T15:21:11.667011+0000","last_active":"2024-04-15T15:21:16.691301+0000","last_peered":"2024-04-15T15:21:16.691301+0000","last_clean":"2024-04-15T15:21:16.691301+0000","last_became_active":"2024-04-15T15:21:11.663992+0000","last_became_peered":"2024-04-15T15:21:11.663992+0000","last_unstale":"2024-04-15T15:21:16.691301+0000","last_undegraded":"2024-04-15T15:21:16.691301+0000","last_fullsized":"2024-04-15T15:21:16.691301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685332+0000","last_change":"2024-04-15T15:21:15.707372+0000","last_active":"2024-04-15T15:21:16.685332+0000","last_peered":"2024-04-15T15:21:16.685332+0000","last_clean":"2024-04-15T15:21:16.685332+0000","last_became_active":"2024-04-15T15:21:15.706978+0000","last_became_peered":"2024-04-15T15:21:15.706978+0000","last_unstale":"2024-04-15T15:21:16.685332+0000","last_undegraded":"2024-04-15T15:21:16.685332+0000","last_fullsized":"2024-04-15T15:21:16.685332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:32.327246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730471+0000","last_change":"2024-04-15T15:21:10.652340+0000","last_active":"2024-04-15T15:21:16.730471+0000","last_peered":"2024-04-15T15:21:16.730471+0000","last_clean":"2024-04-15T15:21:16.730471+0000","last_became_active":"2024-04-15T15:21:10.652016+0000","last_became_peered":"2024-04-15T15:21:10.652016+0000","last_unstale":"2024-04-15T15:21:16.730471+0000","last_undegraded":"2024-04-15T15:21:16.730471+0000","last_fullsized":"2024-04-15T15:21:16.730471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:54:08.208968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691646+0000","last_change":"2024-04-15T15:21:11.663781+0000","last_active":"2024-04-15T15:21:16.691646+0000","last_peered":"2024-04-15T15:21:16.691646+0000","last_clean":"2024-04-15T15:21:16.691646+0000","last_became_active":"2024-04-15T15:21:11.663232+0000","last_became_peered":"2024-04-15T15:21:11.663232+0000","last_unstale":"2024-04-15T15:21:16.691646+0000","last_undegraded":"2024-04-15T15:21:16.691646+0000","last_fullsized":"2024-04-15T15:21:16.691646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.568336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685080+0000","last_change":"2024-04-15T15:21:15.695846+0000","last_active":"2024-04-15T15:21:16.685080+0000","last_peered":"2024-04-15T15:21:16.685080+0000","last_clean":"2024-04-15T15:21:16.685080+0000","last_became_active":"2024-04-15T15:21:15.695426+0000","last_became_peered":"2024-04-15T15:21:15.695426+0000","last_unstale":"2024-04-15T15:21:16.685080+0000","last_undegraded":"2024-04-15T15:21:16.685080+0000","last_fullsized":"2024-04-15T15:21:16.685080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:28.518948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729851+0000","last_change":"2024-04-15T15:21:11.662485+0000","last_active":"2024-04-15T15:21:16.729851+0000","last_peered":"2024-04-15T15:21:16.729851+0000","last_clean":"2024-04-15T15:21:16.729851+0000","last_became_active":"2024-04-15T15:21:11.662183+0000","last_became_peered":"2024-04-15T15:21:11.662183+0000","last_unstale":"2024-04-15T15:21:16.729851+0000","last_undegraded":"2024-04-15T15:21:16.729851+0000","last_fullsized":"2024-04-15T15:21:16.729851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730547+0000","last_change":"2024-04-15T15:21:10.648295+0000","last_active":"2024-04-15T15:21:16.730547+0000","last_peered":"2024-04-15T15:21:16.730547+0000","last_clean":"2024-04-15T15:21:16.730547+0000","last_became_active":"2024-04-15T15:21:10.647985+0000","last_became_peered":"2024-04-15T15:21:10.647985+0000","last_unstale":"2024-04-15T15:21:16.730547+0000","last_undegraded":"2024-04-15T15:21:16.730547+0000","last_fullsized":"2024-04-15T15:21:16.730547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:52.730839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685664+0000","last_change":"2024-04-15T15:21:15.699611+0000","last_active":"2024-04-15T15:21:16.685664+0000","last_peered":"2024-04-15T15:21:16.685664+0000","last_clean":"2024-04-15T15:21:16.685664+0000","last_became_active":"2024-04-15T15:21:15.699129+0000","last_became_peered":"2024-04-15T15:21:15.699129+0000","last_unstale":"2024-04-15T15:21:16.685664+0000","last_undegraded":"2024-04-15T15:21:16.685664+0000","last_fullsized":"2024-04-15T15:21:16.685664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732647+0000","last_change":"2024-04-15T15:21:11.644620+0000","last_active":"2024-04-15T15:21:16.732647+0000","last_peered":"2024-04-15T15:21:16.732647+0000","last_clean":"2024-04-15T15:21:16.732647+0000","last_became_active":"2024-04-15T15:21:11.644273+0000","last_became_peered":"2024-04-15T15:21:11.644273+0000","last_unstale":"2024-04-15T15:21:16.732647+0000","last_undegraded":"2024-04-15T15:21:16.732647+0000","last_fullsized":"2024-04-15T15:21:16.732647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730443+0000","last_change":"2024-04-15T15:21:10.636457+0000","last_active":"2024-04-15T15:21:16.730443+0000","last_peered":"2024-04-15T15:21:16.730443+0000","last_clean":"2024-04-15T15:21:16.730443+0000","last_became_active":"2024-04-15T15:21:10.636143+0000","last_became_peered":"2024-04-15T15:21:10.636143+0000","last_unstale":"2024-04-15T15:21:16.730443+0000","last_undegraded":"2024-04-15T15:21:16.730443+0000","last_fullsized":"2024-04-15T15:21:16.730443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685684+0000","last_change":"2024-04-15T15:21:15.703570+0000","last_active":"2024-04-15T15:21:16.685684+0000","last_peered":"2024-04-15T15:21:16.685684+0000","last_clean":"2024-04-15T15:21:16.685684+0000","last_became_active":"2024-04-15T15:21:15.703066+0000","last_became_peered":"2024-04-15T15:21:15.703066+0000","last_unstale":"2024-04-15T15:21:16.685684+0000","last_undegraded":"2024-04-15T15:21:16.685684+0000","last_fullsized":"2024-04-15T15:21:16.685684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732393+0000","last_change":"2024-04-15T15:21:11.646745+0000","last_active":"2024-04-15T15:21:16.732393+0000","last_peered":"2024-04-15T15:21:16.732393+0000","last_clean":"2024-04-15T15:21:16.732393+0000","last_became_active":"2024-04-15T15:21:11.646375+0000","last_became_peered":"2024-04-15T15:21:11.646375+0000","last_unstale":"2024-04-15T15:21:16.732393+0000","last_undegraded":"2024-04-15T15:21:16.732393+0000","last_fullsized":"2024-04-15T15:21:16.732393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729475+0000","last_change":"2024-04-15T15:21:10.642462+0000","last_active":"2024-04-15T15:21:16.729475+0000","last_peered":"2024-04-15T15:21:16.729475+0000","last_clean":"2024-04-15T15:21:16.729475+0000","last_became_active":"2024-04-15T15:21:10.642168+0000","last_became_peered":"2024-04-15T15:21:10.642168+0000","last_unstale":"2024-04-15T15:21:16.729475+0000","last_undegraded":"2024-04-15T15:21:16.729475+0000","last_fullsized":"2024-04-15T15:21:16.729475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731042+0000","last_change":"2024-04-15T15:21:15.901600+0000","last_active":"2024-04-15T15:21:16.731042+0000","last_peered":"2024-04-15T15:21:16.731042+0000","last_clean":"2024-04-15T15:21:16.731042+0000","last_became_active":"2024-04-15T15:21:15.901233+0000","last_became_peered":"2024-04-15T15:21:15.901233+0000","last_unstale":"2024-04-15T15:21:16.731042+0000","last_undegraded":"2024-04-15T15:21:16.731042+0000","last_fullsized":"2024-04-15T15:21:16.731042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:30.138777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685208+0000","last_change":"2024-04-15T15:21:11.648506+0000","last_active":"2024-04-15T15:21:16.685208+0000","last_peered":"2024-04-15T15:21:16.685208+0000","last_clean":"2024-04-15T15:21:16.685208+0000","last_became_active":"2024-04-15T15:21:11.647972+0000","last_became_peered":"2024-04-15T15:21:11.647972+0000","last_unstale":"2024-04-15T15:21:16.685208+0000","last_undegraded":"2024-04-15T15:21:16.685208+0000","last_fullsized":"2024-04-15T15:21:16.685208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:09.853761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729694+0000","last_change":"2024-04-15T15:21:10.646955+0000","last_active":"2024-04-15T15:21:16.729694+0000","last_peered":"2024-04-15T15:21:16.729694+0000","last_clean":"2024-04-15T15:21:16.729694+0000","last_became_active":"2024-04-15T15:21:10.646308+0000","last_became_peered":"2024-04-15T15:21:10.646308+0000","last_unstale":"2024-04-15T15:21:16.729694+0000","last_undegraded":"2024-04-15T15:21:16.729694+0000","last_fullsized":"2024-04-15T15:21:16.729694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685568+0000","last_change":"2024-04-15T15:21:15.699814+0000","last_active":"2024-04-15T15:21:16.685568+0000","last_peered":"2024-04-15T15:21:16.685568+0000","last_clean":"2024-04-15T15:21:16.685568+0000","last_became_active":"2024-04-15T15:21:15.699341+0000","last_became_peered":"2024-04-15T15:21:15.699341+0000","last_unstale":"2024-04-15T15:21:16.685568+0000","last_undegraded":"2024-04-15T15:21:16.685568+0000","last_fullsized":"2024-04-15T15:21:16.685568+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:51.618858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731527+0000","last_change":"2024-04-15T15:21:11.646950+0000","last_active":"2024-04-15T15:21:16.731527+0000","last_peered":"2024-04-15T15:21:16.731527+0000","last_clean":"2024-04-15T15:21:16.731527+0000","last_became_active":"2024-04-15T15:21:11.646633+0000","last_became_peered":"2024-04-15T15:21:11.646633+0000","last_unstale":"2024-04-15T15:21:16.731527+0000","last_undegraded":"2024-04-15T15:21:16.731527+0000","last_fullsized":"2024-04-15T15:21:16.731527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:56.796018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730414+0000","last_change":"2024-04-15T15:21:10.646693+0000","last_active":"2024-04-15T15:21:16.730414+0000","last_peered":"2024-04-15T15:21:16.730414+0000","last_clean":"2024-04-15T15:21:16.730414+0000","last_became_active":"2024-04-15T15:21:10.646328+0000","last_became_peered":"2024-04-15T15:21:10.646328+0000","last_unstale":"2024-04-15T15:21:16.730414+0000","last_undegraded":"2024-04-15T15:21:16.730414+0000","last_fullsized":"2024-04-15T15:21:16.730414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685127+0000","last_change":"2024-04-15T15:21:15.902524+0000","last_active":"2024-04-15T15:21:16.685127+0000","last_peered":"2024-04-15T15:21:16.685127+0000","last_clean":"2024-04-15T15:21:16.685127+0000","last_became_active":"2024-04-15T15:21:15.901936+0000","last_became_peered":"2024-04-15T15:21:15.901936+0000","last_unstale":"2024-04-15T15:21:16.685127+0000","last_undegraded":"2024-04-15T15:21:16.685127+0000","last_fullsized":"2024-04-15T15:21:16.685127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:59.443259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733036+0000","last_change":"2024-04-15T15:21:11.657367+0000","last_active":"2024-04-15T15:21:16.733036+0000","last_peered":"2024-04-15T15:21:16.733036+0000","last_clean":"2024-04-15T15:21:16.733036+0000","last_became_active":"2024-04-15T15:21:11.657002+0000","last_became_peered":"2024-04-15T15:21:11.657002+0000","last_unstale":"2024-04-15T15:21:16.733036+0000","last_undegraded":"2024-04-15T15:21:16.733036+0000","last_fullsized":"2024-04-15T15:21:16.733036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:02.359534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731010+0000","last_change":"2024-04-15T15:21:10.644925+0000","last_active":"2024-04-15T15:21:16.731010+0000","last_peered":"2024-04-15T15:21:16.731010+0000","last_clean":"2024-04-15T15:21:16.731010+0000","last_became_active":"2024-04-15T15:21:10.644639+0000","last_became_peered":"2024-04-15T15:21:10.644639+0000","last_unstale":"2024-04-15T15:21:16.731010+0000","last_undegraded":"2024-04-15T15:21:16.731010+0000","last_fullsized":"2024-04-15T15:21:16.731010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732150+0000","last_change":"2024-04-15T15:21:15.902755+0000","last_active":"2024-04-15T15:21:16.732150+0000","last_peered":"2024-04-15T15:21:16.732150+0000","last_clean":"2024-04-15T15:21:16.732150+0000","last_became_active":"2024-04-15T15:21:15.902338+0000","last_became_peered":"2024-04-15T15:21:15.902338+0000","last_unstale":"2024-04-15T15:21:16.732150+0000","last_undegraded":"2024-04-15T15:21:16.732150+0000","last_fullsized":"2024-04-15T15:21:16.732150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:13.820849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730647+0000","last_change":"2024-04-15T15:21:11.654427+0000","last_active":"2024-04-15T15:21:16.730647+0000","last_peered":"2024-04-15T15:21:16.730647+0000","last_clean":"2024-04-15T15:21:16.730647+0000","last_became_active":"2024-04-15T15:21:11.654037+0000","last_became_peered":"2024-04-15T15:21:11.654037+0000","last_unstale":"2024-04-15T15:21:16.730647+0000","last_undegraded":"2024-04-15T15:21:16.730647+0000","last_fullsized":"2024-04-15T15:21:16.730647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731058+0000","last_change":"2024-04-15T15:21:10.649235+0000","last_active":"2024-04-15T15:21:16.731058+0000","last_peered":"2024-04-15T15:21:16.731058+0000","last_clean":"2024-04-15T15:21:16.731058+0000","last_became_active":"2024-04-15T15:21:10.648935+0000","last_became_peered":"2024-04-15T15:21:10.648935+0000","last_unstale":"2024-04-15T15:21:16.731058+0000","last_undegraded":"2024-04-15T15:21:16.731058+0000","last_fullsized":"2024-04-15T15:21:16.731058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:55.786608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729854+0000","last_change":"2024-04-15T15:21:15.899286+0000","last_active":"2024-04-15T15:21:16.729854+0000","last_peered":"2024-04-15T15:21:16.729854+0000","last_clean":"2024-04-15T15:21:16.729854+0000","last_became_active":"2024-04-15T15:21:15.898902+0000","last_became_peered":"2024-04-15T15:21:15.898902+0000","last_unstale":"2024-04-15T15:21:16.729854+0000","last_undegraded":"2024-04-15T15:21:16.729854+0000","last_fullsized":"2024-04-15T15:21:16.729854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691740+0000","last_change":"2024-04-15T15:21:11.661691+0000","last_active":"2024-04-15T15:21:16.691740+0000","last_peered":"2024-04-15T15:21:16.691740+0000","last_clean":"2024-04-15T15:21:16.691740+0000","last_became_active":"2024-04-15T15:21:11.661289+0000","last_became_peered":"2024-04-15T15:21:11.661289+0000","last_unstale":"2024-04-15T15:21:16.691740+0000","last_undegraded":"2024-04-15T15:21:16.691740+0000","last_fullsized":"2024-04-15T15:21:16.691740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:08.032185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730825+0000","last_change":"2024-04-15T15:21:10.637826+0000","last_active":"2024-04-15T15:21:16.730825+0000","last_peered":"2024-04-15T15:21:16.730825+0000","last_clean":"2024-04-15T15:21:16.730825+0000","last_became_active":"2024-04-15T15:21:10.637566+0000","last_became_peered":"2024-04-15T15:21:10.637566+0000","last_unstale":"2024-04-15T15:21:16.730825+0000","last_undegraded":"2024-04-15T15:21:16.730825+0000","last_fullsized":"2024-04-15T15:21:16.730825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:31.743027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729641+0000","last_change":"2024-04-15T15:21:15.707987+0000","last_active":"2024-04-15T15:21:16.729641+0000","last_peered":"2024-04-15T15:21:16.729641+0000","last_clean":"2024-04-15T15:21:16.729641+0000","last_became_active":"2024-04-15T15:21:15.707472+0000","last_became_peered":"2024-04-15T15:21:15.707472+0000","last_unstale":"2024-04-15T15:21:16.729641+0000","last_undegraded":"2024-04-15T15:21:16.729641+0000","last_fullsized":"2024-04-15T15:21:16.729641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685621+0000","last_change":"2024-04-15T15:21:11.644409+0000","last_active":"2024-04-15T15:21:16.685621+0000","last_peered":"2024-04-15T15:21:16.685621+0000","last_clean":"2024-04-15T15:21:16.685621+0000","last_became_active":"2024-04-15T15:21:11.644095+0000","last_became_peered":"2024-04-15T15:21:11.644095+0000","last_unstale":"2024-04-15T15:21:16.685621+0000","last_undegraded":"2024-04-15T15:21:16.685621+0000","last_fullsized":"2024-04-15T15:21:16.685621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:32.752437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691377+0000","last_change":"2024-04-15T15:21:10.646053+0000","last_active":"2024-04-15T15:21:16.691377+0000","last_peered":"2024-04-15T15:21:16.691377+0000","last_clean":"2024-04-15T15:21:16.691377+0000","last_became_active":"2024-04-15T15:21:10.645737+0000","last_became_peered":"2024-04-15T15:21:10.645737+0000","last_unstale":"2024-04-15T15:21:16.691377+0000","last_undegraded":"2024-04-15T15:21:16.691377+0000","last_fullsized":"2024-04-15T15:21:16.691377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730648+0000","last_change":"2024-04-15T15:21:15.699348+0000","last_active":"2024-04-15T15:21:16.730648+0000","last_peered":"2024-04-15T15:21:16.730648+0000","last_clean":"2024-04-15T15:21:16.730648+0000","last_became_active":"2024-04-15T15:21:15.698962+0000","last_became_peered":"2024-04-15T15:21:15.698962+0000","last_unstale":"2024-04-15T15:21:16.730648+0000","last_undegraded":"2024-04-15T15:21:16.730648+0000","last_fullsized":"2024-04-15T15:21:16.730648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:35.384335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729484+0000","last_change":"2024-04-15T15:21:11.653613+0000","last_active":"2024-04-15T15:21:16.729484+0000","last_peered":"2024-04-15T15:21:16.729484+0000","last_clean":"2024-04-15T15:21:16.729484+0000","last_became_active":"2024-04-15T15:21:11.653306+0000","last_became_peered":"2024-04-15T15:21:11.653306+0000","last_unstale":"2024-04-15T15:21:16.729484+0000","last_undegraded":"2024-04-15T15:21:16.729484+0000","last_fullsized":"2024-04-15T15:21:16.729484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731231+0000","last_change":"2024-04-15T15:21:10.636526+0000","last_active":"2024-04-15T15:21:16.731231+0000","last_peered":"2024-04-15T15:21:16.731231+0000","last_clean":"2024-04-15T15:21:16.731231+0000","last_became_active":"2024-04-15T15:21:10.636211+0000","last_became_peered":"2024-04-15T15:21:10.636211+0000","last_unstale":"2024-04-15T15:21:16.731231+0000","last_undegraded":"2024-04-15T15:21:16.731231+0000","last_fullsized":"2024-04-15T15:21:16.731231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731153+0000","last_change":"2024-04-15T15:21:15.706536+0000","last_active":"2024-04-15T15:21:16.731153+0000","last_peered":"2024-04-15T15:21:16.731153+0000","last_clean":"2024-04-15T15:21:16.731153+0000","last_became_active":"2024-04-15T15:21:15.706130+0000","last_became_peered":"2024-04-15T15:21:15.706130+0000","last_unstale":"2024-04-15T15:21:16.731153+0000","last_undegraded":"2024-04-15T15:21:16.731153+0000","last_fullsized":"2024-04-15T15:21:16.731153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:35.371338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729702+0000","last_change":"2024-04-15T15:21:11.649636+0000","last_active":"2024-04-15T15:21:16.729702+0000","last_peered":"2024-04-15T15:21:16.729702+0000","last_clean":"2024-04-15T15:21:16.729702+0000","last_became_active":"2024-04-15T15:21:11.649334+0000","last_became_peered":"2024-04-15T15:21:11.649334+0000","last_unstale":"2024-04-15T15:21:16.729702+0000","last_undegraded":"2024-04-15T15:21:16.729702+0000","last_fullsized":"2024-04-15T15:21:16.729702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:01:17.255107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730694+0000","last_change":"2024-04-15T15:21:10.647209+0000","last_active":"2024-04-15T15:21:16.730694+0000","last_peered":"2024-04-15T15:21:16.730694+0000","last_clean":"2024-04-15T15:21:16.730694+0000","last_became_active":"2024-04-15T15:21:10.646650+0000","last_became_peered":"2024-04-15T15:21:10.646650+0000","last_unstale":"2024-04-15T15:21:16.730694+0000","last_undegraded":"2024-04-15T15:21:16.730694+0000","last_fullsized":"2024-04-15T15:21:16.730694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:31.743027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731537+0000","last_change":"2024-04-15T15:21:15.901283+0000","last_active":"2024-04-15T15:21:16.731537+0000","last_peered":"2024-04-15T15:21:16.731537+0000","last_clean":"2024-04-15T15:21:16.731537+0000","last_became_active":"2024-04-15T15:21:15.900931+0000","last_became_peered":"2024-04-15T15:21:15.900931+0000","last_unstale":"2024-04-15T15:21:16.731537+0000","last_undegraded":"2024-04-15T15:21:16.731537+0000","last_fullsized":"2024-04-15T15:21:16.731537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:35.371338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732331+0000","last_change":"2024-04-15T15:21:11.664990+0000","last_active":"2024-04-15T15:21:16.732331+0000","last_peered":"2024-04-15T15:21:16.732331+0000","last_clean":"2024-04-15T15:21:16.732331+0000","last_became_active":"2024-04-15T15:21:11.664706+0000","last_became_peered":"2024-04-15T15:21:11.664706+0000","last_unstale":"2024-04-15T15:21:16.732331+0000","last_undegraded":"2024-04-15T15:21:16.732331+0000","last_fullsized":"2024-04-15T15:21:16.732331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691196+0000","last_change":"2024-04-15T15:21:10.645516+0000","last_active":"2024-04-15T15:21:16.691196+0000","last_peered":"2024-04-15T15:21:16.691196+0000","last_clean":"2024-04-15T15:21:16.691196+0000","last_became_active":"2024-04-15T15:21:10.645191+0000","last_became_peered":"2024-04-15T15:21:10.645191+0000","last_unstale":"2024-04-15T15:21:16.691196+0000","last_undegraded":"2024-04-15T15:21:16.691196+0000","last_fullsized":"2024-04-15T15:21:16.691196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730801+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:21:16.730801+0000","last_peered":"2024-04-15T15:21:16.730801+0000","last_clean":"2024-04-15T15:21:16.730801+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:21:16.730801+0000","last_undegraded":"2024-04-15T15:21:16.730801+0000","last_fullsized":"2024-04-15T15:21:16.730801+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730072+0000","last_change":"2024-04-15T15:21:15.901627+0000","last_active":"2024-04-15T15:21:16.730072+0000","last_peered":"2024-04-15T15:21:16.730072+0000","last_clean":"2024-04-15T15:21:16.730072+0000","last_became_active":"2024-04-15T15:21:15.901132+0000","last_became_peered":"2024-04-15T15:21:15.901132+0000","last_unstale":"2024-04-15T15:21:16.730072+0000","last_undegraded":"2024-04-15T15:21:16.730072+0000","last_fullsized":"2024-04-15T15:21:16.730072+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730781+0000","last_change":"2024-04-15T15:21:11.665709+0000","last_active":"2024-04-15T15:21:16.730781+0000","last_peered":"2024-04-15T15:21:16.730781+0000","last_clean":"2024-04-15T15:21:16.730781+0000","last_became_active":"2024-04-15T15:21:11.665254+0000","last_became_peered":"2024-04-15T15:21:11.665254+0000","last_unstale":"2024-04-15T15:21:16.730781+0000","last_undegraded":"2024-04-15T15:21:16.730781+0000","last_fullsized":"2024-04-15T15:21:16.730781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:31.333813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730984+0000","last_change":"2024-04-15T15:21:10.644131+0000","last_active":"2024-04-15T15:21:16.730984+0000","last_peered":"2024-04-15T15:21:16.730984+0000","last_clean":"2024-04-15T15:21:16.730984+0000","last_became_active":"2024-04-15T15:21:10.643844+0000","last_became_peered":"2024-04-15T15:21:10.643844+0000","last_unstale":"2024-04-15T15:21:16.730984+0000","last_undegraded":"2024-04-15T15:21:16.730984+0000","last_fullsized":"2024-04-15T15:21:16.730984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:16:25.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729900+0000","last_change":"2024-04-15T15:21:15.899284+0000","last_active":"2024-04-15T15:21:16.729900+0000","last_peered":"2024-04-15T15:21:16.729900+0000","last_clean":"2024-04-15T15:21:16.729900+0000","last_became_active":"2024-04-15T15:21:15.898822+0000","last_became_peered":"2024-04-15T15:21:15.898822+0000","last_unstale":"2024-04-15T15:21:16.729900+0000","last_undegraded":"2024-04-15T15:21:16.729900+0000","last_fullsized":"2024-04-15T15:21:16.729900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:24.117458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691307+0000","last_change":"2024-04-15T15:21:11.663221+0000","last_active":"2024-04-15T15:21:16.691307+0000","last_peered":"2024-04-15T15:21:16.691307+0000","last_clean":"2024-04-15T15:21:16.691307+0000","last_became_active":"2024-04-15T15:21:11.662827+0000","last_became_peered":"2024-04-15T15:21:11.662827+0000","last_unstale":"2024-04-15T15:21:16.691307+0000","last_undegraded":"2024-04-15T15:21:16.691307+0000","last_fullsized":"2024-04-15T15:21:16.691307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:15.476006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730374+0000","last_change":"2024-04-15T15:21:10.636999+0000","last_active":"2024-04-15T15:21:16.730374+0000","last_peered":"2024-04-15T15:21:16.730374+0000","last_clean":"2024-04-15T15:21:16.730374+0000","last_became_active":"2024-04-15T15:21:10.636729+0000","last_became_peered":"2024-04-15T15:21:10.636729+0000","last_unstale":"2024-04-15T15:21:16.730374+0000","last_undegraded":"2024-04-15T15:21:16.730374+0000","last_fullsized":"2024-04-15T15:21:16.730374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730116+0000","last_change":"2024-04-15T15:21:15.708002+0000","last_active":"2024-04-15T15:21:16.730116+0000","last_peered":"2024-04-15T15:21:16.730116+0000","last_clean":"2024-04-15T15:21:16.730116+0000","last_became_active":"2024-04-15T15:21:15.707636+0000","last_became_peered":"2024-04-15T15:21:15.707636+0000","last_unstale":"2024-04-15T15:21:16.730116+0000","last_undegraded":"2024-04-15T15:21:16.730116+0000","last_fullsized":"2024-04-15T15:21:16.730116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730705+0000","last_change":"2024-04-15T15:21:11.662157+0000","last_active":"2024-04-15T15:21:16.730705+0000","last_peered":"2024-04-15T15:21:16.730705+0000","last_clean":"2024-04-15T15:21:16.730705+0000","last_became_active":"2024-04-15T15:21:11.661885+0000","last_became_peered":"2024-04-15T15:21:11.661885+0000","last_unstale":"2024-04-15T15:21:16.730705+0000","last_undegraded":"2024-04-15T15:21:16.730705+0000","last_fullsized":"2024-04-15T15:21:16.730705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:30.561749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731734+0000","last_change":"2024-04-15T15:21:10.635657+0000","last_active":"2024-04-15T15:21:16.731734+0000","last_peered":"2024-04-15T15:21:16.731734+0000","last_clean":"2024-04-15T15:21:16.731734+0000","last_became_active":"2024-04-15T15:21:10.635339+0000","last_became_peered":"2024-04-15T15:21:10.635339+0000","last_unstale":"2024-04-15T15:21:16.731734+0000","last_undegraded":"2024-04-15T15:21:16.731734+0000","last_fullsized":"2024-04-15T15:21:16.731734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690911+0000","last_change":"2024-04-15T15:21:15.902642+0000","last_active":"2024-04-15T15:21:16.690911+0000","last_peered":"2024-04-15T15:21:16.690911+0000","last_clean":"2024-04-15T15:21:16.690911+0000","last_became_active":"2024-04-15T15:21:15.902252+0000","last_became_peered":"2024-04-15T15:21:15.902252+0000","last_unstale":"2024-04-15T15:21:16.690911+0000","last_undegraded":"2024-04-15T15:21:16.690911+0000","last_fullsized":"2024-04-15T15:21:16.690911+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:13.820849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730900+0000","last_change":"2024-04-15T15:21:11.658612+0000","last_active":"2024-04-15T15:21:16.730900+0000","last_peered":"2024-04-15T15:21:16.730900+0000","last_clean":"2024-04-15T15:21:16.730900+0000","last_became_active":"2024-04-15T15:21:11.658312+0000","last_became_peered":"2024-04-15T15:21:11.658312+0000","last_unstale":"2024-04-15T15:21:16.730900+0000","last_undegraded":"2024-04-15T15:21:16.730900+0000","last_fullsized":"2024-04-15T15:21:16.730900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:15.476006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685731+0000","last_change":"2024-04-15T15:21:10.636964+0000","last_active":"2024-04-15T15:21:16.685731+0000","last_peered":"2024-04-15T15:21:16.685731+0000","last_clean":"2024-04-15T15:21:16.685731+0000","last_became_active":"2024-04-15T15:21:10.636641+0000","last_became_peered":"2024-04-15T15:21:10.636641+0000","last_unstale":"2024-04-15T15:21:16.685731+0000","last_undegraded":"2024-04-15T15:21:16.685731+0000","last_fullsized":"2024-04-15T15:21:16.685731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:27:05.012468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.684974+0000","last_change":"2024-04-15T15:21:15.706953+0000","last_active":"2024-04-15T15:21:16.684974+0000","last_peered":"2024-04-15T15:21:16.684974+0000","last_clean":"2024-04-15T15:21:16.684974+0000","last_became_active":"2024-04-15T15:21:15.706465+0000","last_became_peered":"2024-04-15T15:21:15.706465+0000","last_unstale":"2024-04-15T15:21:16.684974+0000","last_undegraded":"2024-04-15T15:21:16.684974+0000","last_fullsized":"2024-04-15T15:21:16.684974+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:06.410056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731056+0000","last_change":"2024-04-15T15:21:11.658373+0000","last_active":"2024-04-15T15:21:16.731056+0000","last_peered":"2024-04-15T15:21:16.731056+0000","last_clean":"2024-04-15T15:21:16.731056+0000","last_became_active":"2024-04-15T15:21:11.658069+0000","last_became_peered":"2024-04-15T15:21:11.658069+0000","last_unstale":"2024-04-15T15:21:16.731056+0000","last_undegraded":"2024-04-15T15:21:16.731056+0000","last_fullsized":"2024-04-15T15:21:16.731056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:31.320816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730149+0000","last_change":"2024-04-15T15:21:10.637607+0000","last_active":"2024-04-15T15:21:16.730149+0000","last_peered":"2024-04-15T15:21:16.730149+0000","last_clean":"2024-04-15T15:21:16.730149+0000","last_became_active":"2024-04-15T15:21:10.637277+0000","last_became_peered":"2024-04-15T15:21:10.637277+0000","last_unstale":"2024-04-15T15:21:16.730149+0000","last_undegraded":"2024-04-15T15:21:16.730149+0000","last_fullsized":"2024-04-15T15:21:16.730149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:03:48.192888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731366+0000","last_change":"2024-04-15T15:21:15.696964+0000","last_active":"2024-04-15T15:21:16.731366+0000","last_peered":"2024-04-15T15:21:16.731366+0000","last_clean":"2024-04-15T15:21:16.731366+0000","last_became_active":"2024-04-15T15:21:15.696559+0000","last_became_peered":"2024-04-15T15:21:15.696559+0000","last_unstale":"2024-04-15T15:21:16.731366+0000","last_undegraded":"2024-04-15T15:21:16.731366+0000","last_fullsized":"2024-04-15T15:21:16.731366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:35.384335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729725+0000","last_change":"2024-04-15T15:21:11.661036+0000","last_active":"2024-04-15T15:21:16.729725+0000","last_peered":"2024-04-15T15:21:16.729725+0000","last_clean":"2024-04-15T15:21:16.729725+0000","last_became_active":"2024-04-15T15:21:11.660616+0000","last_became_peered":"2024-04-15T15:21:11.660616+0000","last_unstale":"2024-04-15T15:21:16.729725+0000","last_undegraded":"2024-04-15T15:21:16.729725+0000","last_fullsized":"2024-04-15T15:21:16.729725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729989+0000","last_change":"2024-04-15T15:21:10.644599+0000","last_active":"2024-04-15T15:21:16.729989+0000","last_peered":"2024-04-15T15:21:16.729989+0000","last_clean":"2024-04-15T15:21:16.729989+0000","last_became_active":"2024-04-15T15:21:10.644274+0000","last_became_peered":"2024-04-15T15:21:10.644274+0000","last_unstale":"2024-04-15T15:21:16.729989+0000","last_undegraded":"2024-04-15T15:21:16.729989+0000","last_fullsized":"2024-04-15T15:21:16.729989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732952+0000","last_change":"2024-04-15T15:21:15.901481+0000","last_active":"2024-04-15T15:21:16.732952+0000","last_peered":"2024-04-15T15:21:16.732952+0000","last_clean":"2024-04-15T15:21:16.732952+0000","last_became_active":"2024-04-15T15:21:15.901096+0000","last_became_peered":"2024-04-15T15:21:15.901096+0000","last_unstale":"2024-04-15T15:21:16.732952+0000","last_undegraded":"2024-04-15T15:21:16.732952+0000","last_fullsized":"2024-04-15T15:21:16.732952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:38.490295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730312+0000","last_change":"2024-04-15T15:21:11.658341+0000","last_active":"2024-04-15T15:21:16.730312+0000","last_peered":"2024-04-15T15:21:16.730312+0000","last_clean":"2024-04-15T15:21:16.730312+0000","last_became_active":"2024-04-15T15:21:11.658024+0000","last_became_peered":"2024-04-15T15:21:11.658024+0000","last_unstale":"2024-04-15T15:21:16.730312+0000","last_undegraded":"2024-04-15T15:21:16.730312+0000","last_fullsized":"2024-04-15T15:21:16.730312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.568336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691308+0000","last_change":"2024-04-15T15:21:10.645818+0000","last_active":"2024-04-15T15:21:16.691308+0000","last_peered":"2024-04-15T15:21:16.691308+0000","last_clean":"2024-04-15T15:21:16.691308+0000","last_became_active":"2024-04-15T15:21:10.645371+0000","last_became_peered":"2024-04-15T15:21:10.645371+0000","last_unstale":"2024-04-15T15:21:16.691308+0000","last_undegraded":"2024-04-15T15:21:16.691308+0000","last_fullsized":"2024-04-15T15:21:16.691308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733065+0000","last_change":"2024-04-15T15:21:15.899911+0000","last_active":"2024-04-15T15:21:16.733065+0000","last_peered":"2024-04-15T15:21:16.733065+0000","last_clean":"2024-04-15T15:21:16.733065+0000","last_became_active":"2024-04-15T15:21:15.899428+0000","last_became_peered":"2024-04-15T15:21:15.899428+0000","last_unstale":"2024-04-15T15:21:16.733065+0000","last_undegraded":"2024-04-15T15:21:16.733065+0000","last_fullsized":"2024-04-15T15:21:16.733065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:18:40.268133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731100+0000","last_change":"2024-04-15T15:21:11.659032+0000","last_active":"2024-04-15T15:21:16.731100+0000","last_peered":"2024-04-15T15:21:16.731100+0000","last_clean":"2024-04-15T15:21:16.731100+0000","last_became_active":"2024-04-15T15:21:11.658769+0000","last_became_peered":"2024-04-15T15:21:11.658769+0000","last_unstale":"2024-04-15T15:21:16.731100+0000","last_undegraded":"2024-04-15T15:21:16.731100+0000","last_fullsized":"2024-04-15T15:21:16.731100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730358+0000","last_change":"2024-04-15T15:21:10.640715+0000","last_active":"2024-04-15T15:21:16.730358+0000","last_peered":"2024-04-15T15:21:16.730358+0000","last_clean":"2024-04-15T15:21:16.730358+0000","last_became_active":"2024-04-15T15:21:10.640364+0000","last_became_peered":"2024-04-15T15:21:10.640364+0000","last_unstale":"2024-04-15T15:21:16.730358+0000","last_undegraded":"2024-04-15T15:21:16.730358+0000","last_fullsized":"2024-04-15T15:21:16.730358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:54:53.368896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691556+0000","last_change":"2024-04-15T15:21:15.901574+0000","last_active":"2024-04-15T15:21:16.691556+0000","last_peered":"2024-04-15T15:21:16.691556+0000","last_clean":"2024-04-15T15:21:16.691556+0000","last_became_active":"2024-04-15T15:21:15.901144+0000","last_became_peered":"2024-04-15T15:21:15.901144+0000","last_unstale":"2024-04-15T15:21:16.691556+0000","last_undegraded":"2024-04-15T15:21:16.691556+0000","last_fullsized":"2024-04-15T15:21:16.691556+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:34.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730711+0000","last_change":"2024-04-15T15:21:11.655252+0000","last_active":"2024-04-15T15:21:16.730711+0000","last_peered":"2024-04-15T15:21:16.730711+0000","last_clean":"2024-04-15T15:21:16.730711+0000","last_became_active":"2024-04-15T15:21:11.654839+0000","last_became_peered":"2024-04-15T15:21:11.654839+0000","last_unstale":"2024-04-15T15:21:16.730711+0000","last_undegraded":"2024-04-15T15:21:16.730711+0000","last_fullsized":"2024-04-15T15:21:16.730711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:08.032185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:21.251862+0000","last_change":"2024-04-15T15:21:10.645952+0000","last_active":"2024-04-15T15:21:21.251862+0000","last_peered":"2024-04-15T15:21:21.251862+0000","last_clean":"2024-04-15T15:21:21.251862+0000","last_became_active":"2024-04-15T15:21:10.645618+0000","last_became_peered":"2024-04-15T15:21:10.645618+0000","last_unstale":"2024-04-15T15:21:21.251862+0000","last_undegraded":"2024-04-15T15:21:21.251862+0000","last_fullsized":"2024-04-15T15:21:21.251862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730096+0000","last_change":"2024-04-15T15:21:15.703360+0000","last_active":"2024-04-15T15:21:16.730096+0000","last_peered":"2024-04-15T15:21:16.730096+0000","last_clean":"2024-04-15T15:21:16.730096+0000","last_became_active":"2024-04-15T15:21:15.702868+0000","last_became_peered":"2024-04-15T15:21:15.702868+0000","last_unstale":"2024-04-15T15:21:16.730096+0000","last_undegraded":"2024-04-15T15:21:16.730096+0000","last_fullsized":"2024-04-15T15:21:16.730096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685743+0000","last_change":"2024-04-15T15:21:11.659799+0000","last_active":"2024-04-15T15:21:16.685743+0000","last_peered":"2024-04-15T15:21:16.685743+0000","last_clean":"2024-04-15T15:21:16.685743+0000","last_became_active":"2024-04-15T15:21:11.659521+0000","last_became_peered":"2024-04-15T15:21:11.659521+0000","last_unstale":"2024-04-15T15:21:16.685743+0000","last_undegraded":"2024-04-15T15:21:16.685743+0000","last_fullsized":"2024-04-15T15:21:16.685743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:38.472181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730267+0000","last_change":"2024-04-15T15:21:10.645938+0000","last_active":"2024-04-15T15:21:16.730267+0000","last_peered":"2024-04-15T15:21:16.730267+0000","last_clean":"2024-04-15T15:21:16.730267+0000","last_became_active":"2024-04-15T15:21:10.645642+0000","last_became_peered":"2024-04-15T15:21:10.645642+0000","last_unstale":"2024-04-15T15:21:16.730267+0000","last_undegraded":"2024-04-15T15:21:16.730267+0000","last_fullsized":"2024-04-15T15:21:16.730267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685481+0000","last_change":"2024-04-15T15:21:15.705076+0000","last_active":"2024-04-15T15:21:16.685481+0000","last_peered":"2024-04-15T15:21:16.685481+0000","last_clean":"2024-04-15T15:21:16.685481+0000","last_became_active":"2024-04-15T15:21:15.704627+0000","last_became_peered":"2024-04-15T15:21:15.704627+0000","last_unstale":"2024-04-15T15:21:16.685481+0000","last_undegraded":"2024-04-15T15:21:16.685481+0000","last_fullsized":"2024-04-15T15:21:16.685481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:24.117458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731387+0000","last_change":"2024-04-15T15:21:11.665544+0000","last_active":"2024-04-15T15:21:16.731387+0000","last_peered":"2024-04-15T15:21:16.731387+0000","last_clean":"2024-04-15T15:21:16.731387+0000","last_became_active":"2024-04-15T15:21:11.665263+0000","last_became_peered":"2024-04-15T15:21:11.665263+0000","last_unstale":"2024-04-15T15:21:16.731387+0000","last_undegraded":"2024-04-15T15:21:16.731387+0000","last_fullsized":"2024-04-15T15:21:16.731387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691599+0000","last_change":"2024-04-15T15:21:10.648496+0000","last_active":"2024-04-15T15:21:16.691599+0000","last_peered":"2024-04-15T15:21:16.691599+0000","last_clean":"2024-04-15T15:21:16.691599+0000","last_became_active":"2024-04-15T15:21:10.648222+0000","last_became_peered":"2024-04-15T15:21:10.648222+0000","last_unstale":"2024-04-15T15:21:16.691599+0000","last_undegraded":"2024-04-15T15:21:16.691599+0000","last_fullsized":"2024-04-15T15:21:16.691599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732523+0000","last_change":"2024-04-15T15:21:15.699583+0000","last_active":"2024-04-15T15:21:16.732523+0000","last_peered":"2024-04-15T15:21:16.732523+0000","last_clean":"2024-04-15T15:21:16.732523+0000","last_became_active":"2024-04-15T15:21:15.699119+0000","last_became_peered":"2024-04-15T15:21:15.699119+0000","last_unstale":"2024-04-15T15:21:16.732523+0000","last_undegraded":"2024-04-15T15:21:16.732523+0000","last_fullsized":"2024-04-15T15:21:16.732523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:41:57.140691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691502+0000","last_change":"2024-04-15T15:21:11.663796+0000","last_active":"2024-04-15T15:21:16.691502+0000","last_peered":"2024-04-15T15:21:16.691502+0000","last_clean":"2024-04-15T15:21:16.691502+0000","last_became_active":"2024-04-15T15:21:11.663502+0000","last_became_peered":"2024-04-15T15:21:11.663502+0000","last_unstale":"2024-04-15T15:21:16.691502+0000","last_undegraded":"2024-04-15T15:21:16.691502+0000","last_fullsized":"2024-04-15T15:21:16.691502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:31.333813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730207+0000","last_change":"2024-04-15T15:21:10.651176+0000","last_active":"2024-04-15T15:21:16.730207+0000","last_peered":"2024-04-15T15:21:16.730207+0000","last_clean":"2024-04-15T15:21:16.730207+0000","last_became_active":"2024-04-15T15:21:10.650301+0000","last_became_peered":"2024-04-15T15:21:10.650301+0000","last_unstale":"2024-04-15T15:21:16.730207+0000","last_undegraded":"2024-04-15T15:21:16.730207+0000","last_fullsized":"2024-04-15T15:21:16.730207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732313+0000","last_change":"2024-04-15T15:21:15.899209+0000","last_active":"2024-04-15T15:21:16.732313+0000","last_peered":"2024-04-15T15:21:16.732313+0000","last_clean":"2024-04-15T15:21:16.732313+0000","last_became_active":"2024-04-15T15:21:15.898799+0000","last_became_peered":"2024-04-15T15:21:15.898799+0000","last_unstale":"2024-04-15T15:21:16.732313+0000","last_undegraded":"2024-04-15T15:21:16.732313+0000","last_fullsized":"2024-04-15T15:21:16.732313+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:16.350687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731404+0000","last_change":"2024-04-15T15:21:15.902104+0000","last_active":"2024-04-15T15:21:16.731404+0000","last_peered":"2024-04-15T15:21:16.731404+0000","last_clean":"2024-04-15T15:21:16.731404+0000","last_became_active":"2024-04-15T15:21:15.901711+0000","last_became_peered":"2024-04-15T15:21:15.901711+0000","last_unstale":"2024-04-15T15:21:16.731404+0000","last_undegraded":"2024-04-15T15:21:16.731404+0000","last_fullsized":"2024-04-15T15:21:16.731404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:30.138777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729279+0000","last_change":"2024-04-15T15:21:11.649924+0000","last_active":"2024-04-15T15:21:16.729279+0000","last_peered":"2024-04-15T15:21:16.729279+0000","last_clean":"2024-04-15T15:21:16.729279+0000","last_became_active":"2024-04-15T15:21:11.649654+0000","last_became_peered":"2024-04-15T15:21:11.649654+0000","last_unstale":"2024-04-15T15:21:16.729279+0000","last_undegraded":"2024-04-15T15:21:16.729279+0000","last_fullsized":"2024-04-15T15:21:16.729279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691073+0000","last_change":"2024-04-15T15:21:10.647906+0000","last_active":"2024-04-15T15:21:16.691073+0000","last_peered":"2024-04-15T15:21:16.691073+0000","last_clean":"2024-04-15T15:21:16.691073+0000","last_became_active":"2024-04-15T15:21:10.647594+0000","last_became_peered":"2024-04-15T15:21:10.647594+0000","last_unstale":"2024-04-15T15:21:16.691073+0000","last_undegraded":"2024-04-15T15:21:16.691073+0000","last_fullsized":"2024-04-15T15:21:16.691073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729962+0000","last_change":"2024-04-15T15:21:15.707054+0000","last_active":"2024-04-15T15:21:16.729962+0000","last_peered":"2024-04-15T15:21:16.729962+0000","last_clean":"2024-04-15T15:21:16.729962+0000","last_became_active":"2024-04-15T15:21:15.706636+0000","last_became_peered":"2024-04-15T15:21:15.706636+0000","last_unstale":"2024-04-15T15:21:16.729962+0000","last_undegraded":"2024-04-15T15:21:16.729962+0000","last_fullsized":"2024-04-15T15:21:16.729962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:51.618858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732275+0000","last_change":"2024-04-15T15:21:11.644995+0000","last_active":"2024-04-15T15:21:16.732275+0000","last_peered":"2024-04-15T15:21:16.732275+0000","last_clean":"2024-04-15T15:21:16.732275+0000","last_became_active":"2024-04-15T15:21:11.644654+0000","last_became_peered":"2024-04-15T15:21:11.644654+0000","last_unstale":"2024-04-15T15:21:16.732275+0000","last_undegraded":"2024-04-15T15:21:16.732275+0000","last_fullsized":"2024-04-15T15:21:16.732275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732479+0000","last_change":"2024-04-15T15:21:10.645892+0000","last_active":"2024-04-15T15:21:16.732479+0000","last_peered":"2024-04-15T15:21:16.732479+0000","last_clean":"2024-04-15T15:21:16.732479+0000","last_became_active":"2024-04-15T15:21:10.645615+0000","last_became_peered":"2024-04-15T15:21:10.645615+0000","last_unstale":"2024-04-15T15:21:16.732479+0000","last_undegraded":"2024-04-15T15:21:16.732479+0000","last_fullsized":"2024-04-15T15:21:16.732479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:16:25.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730156+0000","last_change":"2024-04-15T15:21:10.646124+0000","last_active":"2024-04-15T15:21:16.730156+0000","last_peered":"2024-04-15T15:21:16.730156+0000","last_clean":"2024-04-15T15:21:16.730156+0000","last_became_active":"2024-04-15T15:21:10.644818+0000","last_became_peered":"2024-04-15T15:21:10.644818+0000","last_unstale":"2024-04-15T15:21:16.730156+0000","last_undegraded":"2024-04-15T15:21:16.730156+0000","last_fullsized":"2024-04-15T15:21:16.730156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730262+0000","last_change":"2024-04-15T15:21:11.648824+0000","last_active":"2024-04-15T15:21:16.730262+0000","last_peered":"2024-04-15T15:21:16.730262+0000","last_clean":"2024-04-15T15:21:16.730262+0000","last_became_active":"2024-04-15T15:21:11.648490+0000","last_became_peered":"2024-04-15T15:21:11.648490+0000","last_unstale":"2024-04-15T15:21:16.730262+0000","last_undegraded":"2024-04-15T15:21:16.730262+0000","last_fullsized":"2024-04-15T15:21:16.730262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730777+0000","last_change":"2024-04-15T15:21:15.699586+0000","last_active":"2024-04-15T15:21:16.730777+0000","last_peered":"2024-04-15T15:21:16.730777+0000","last_clean":"2024-04-15T15:21:16.730777+0000","last_became_active":"2024-04-15T15:21:15.699130+0000","last_became_peered":"2024-04-15T15:21:15.699130+0000","last_unstale":"2024-04-15T15:21:16.730777+0000","last_undegraded":"2024-04-15T15:21:16.730777+0000","last_fullsized":"2024-04-15T15:21:16.730777+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:47.552269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732768+0000","last_change":"2024-04-15T15:21:10.641801+0000","last_active":"2024-04-15T15:21:16.732768+0000","last_peered":"2024-04-15T15:21:16.732768+0000","last_clean":"2024-04-15T15:21:16.732768+0000","last_became_active":"2024-04-15T15:21:10.641511+0000","last_became_peered":"2024-04-15T15:21:10.641511+0000","last_unstale":"2024-04-15T15:21:16.732768+0000","last_undegraded":"2024-04-15T15:21:16.732768+0000","last_fullsized":"2024-04-15T15:21:16.732768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:03:48.192888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685152+0000","last_change":"2024-04-15T15:21:11.650188+0000","last_active":"2024-04-15T15:21:16.685152+0000","last_peered":"2024-04-15T15:21:16.685152+0000","last_clean":"2024-04-15T15:21:16.685152+0000","last_became_active":"2024-04-15T15:21:11.649838+0000","last_became_peered":"2024-04-15T15:21:11.649838+0000","last_unstale":"2024-04-15T15:21:16.685152+0000","last_undegraded":"2024-04-15T15:21:16.685152+0000","last_fullsized":"2024-04-15T15:21:16.685152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:12.989671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732888+0000","last_change":"2024-04-15T15:21:15.699639+0000","last_active":"2024-04-15T15:21:16.732888+0000","last_peered":"2024-04-15T15:21:16.732888+0000","last_clean":"2024-04-15T15:21:16.732888+0000","last_became_active":"2024-04-15T15:21:15.699252+0000","last_became_peered":"2024-04-15T15:21:15.699252+0000","last_unstale":"2024-04-15T15:21:16.732888+0000","last_undegraded":"2024-04-15T15:21:16.732888+0000","last_fullsized":"2024-04-15T15:21:16.732888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:02.471663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729618+0000","last_change":"2024-04-15T15:21:11.662366+0000","last_active":"2024-04-15T15:21:16.729618+0000","last_peered":"2024-04-15T15:21:16.729618+0000","last_clean":"2024-04-15T15:21:16.729618+0000","last_became_active":"2024-04-15T15:21:11.662082+0000","last_became_peered":"2024-04-15T15:21:11.662082+0000","last_unstale":"2024-04-15T15:21:16.729618+0000","last_undegraded":"2024-04-15T15:21:16.729618+0000","last_fullsized":"2024-04-15T15:21:16.729618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731145+0000","last_change":"2024-04-15T15:21:11.667867+0000","last_active":"2024-04-15T15:21:16.731145+0000","last_peered":"2024-04-15T15:21:16.731145+0000","last_clean":"2024-04-15T15:21:16.731145+0000","last_became_active":"2024-04-15T15:21:11.667664+0000","last_became_peered":"2024-04-15T15:21:11.667664+0000","last_unstale":"2024-04-15T15:21:16.731145+0000","last_undegraded":"2024-04-15T15:21:16.731145+0000","last_fullsized":"2024-04-15T15:21:16.731145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:32.752437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691237+0000","last_change":"2024-04-15T15:21:11.662949+0000","last_active":"2024-04-15T15:21:16.691237+0000","last_peered":"2024-04-15T15:21:16.691237+0000","last_clean":"2024-04-15T15:21:16.691237+0000","last_became_active":"2024-04-15T15:21:11.662014+0000","last_became_peered":"2024-04-15T15:21:11.662014+0000","last_unstale":"2024-04-15T15:21:16.691237+0000","last_undegraded":"2024-04-15T15:21:16.691237+0000","last_fullsized":"2024-04-15T15:21:16.691237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:20.066936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730170+0000","last_change":"2024-04-15T15:21:11.643845+0000","last_active":"2024-04-15T15:21:16.730170+0000","last_peered":"2024-04-15T15:21:16.730170+0000","last_clean":"2024-04-15T15:21:16.730170+0000","last_became_active":"2024-04-15T15:21:11.643459+0000","last_became_peered":"2024-04-15T15:21:11.643459+0000","last_unstale":"2024-04-15T15:21:16.730170+0000","last_undegraded":"2024-04-15T15:21:16.730170+0000","last_fullsized":"2024-04-15T15:21:16.730170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:43.501747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733169+0000","last_change":"2024-04-15T15:21:11.649864+0000","last_active":"2024-04-15T15:21:16.733169+0000","last_peered":"2024-04-15T15:21:16.733169+0000","last_clean":"2024-04-15T15:21:16.733169+0000","last_became_active":"2024-04-15T15:21:11.649174+0000","last_became_peered":"2024-04-15T15:21:11.649174+0000","last_unstale":"2024-04-15T15:21:16.733169+0000","last_undegraded":"2024-04-15T15:21:16.733169+0000","last_fullsized":"2024-04-15T15:21:16.733169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:55.392737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730146+0000","last_change":"2024-04-15T15:21:11.649917+0000","last_active":"2024-04-15T15:21:16.730146+0000","last_peered":"2024-04-15T15:21:16.730146+0000","last_clean":"2024-04-15T15:21:16.730146+0000","last_became_active":"2024-04-15T15:21:11.649627+0000","last_became_peered":"2024-04-15T15:21:11.649627+0000","last_unstale":"2024-04-15T15:21:16.730146+0000","last_undegraded":"2024-04-15T15:21:16.730146+0000","last_fullsized":"2024-04-15T15:21:16.730146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730543+0000","last_change":"2024-04-15T15:21:11.666666+0000","last_active":"2024-04-15T15:21:16.730543+0000","last_peered":"2024-04-15T15:21:16.730543+0000","last_clean":"2024-04-15T15:21:16.730543+0000","last_became_active":"2024-04-15T15:21:11.666354+0000","last_became_peered":"2024-04-15T15:21:11.666354+0000","last_unstale":"2024-04-15T15:21:16.730543+0000","last_undegraded":"2024-04-15T15:21:16.730543+0000","last_fullsized":"2024-04-15T15:21:16.730543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732965+0000","last_change":"2024-04-15T15:21:11.649875+0000","last_active":"2024-04-15T15:21:16.732965+0000","last_peered":"2024-04-15T15:21:16.732965+0000","last_clean":"2024-04-15T15:21:16.732965+0000","last_became_active":"2024-04-15T15:21:11.649492+0000","last_became_peered":"2024-04-15T15:21:11.649492+0000","last_unstale":"2024-04-15T15:21:16.732965+0000","last_undegraded":"2024-04-15T15:21:16.732965+0000","last_fullsized":"2024-04-15T15:21:16.732965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690980+0000","last_change":"2024-04-15T15:21:11.663545+0000","last_active":"2024-04-15T15:21:16.690980+0000","last_peered":"2024-04-15T15:21:16.690980+0000","last_clean":"2024-04-15T15:21:16.690980+0000","last_became_active":"2024-04-15T15:21:11.662983+0000","last_became_peered":"2024-04-15T15:21:11.662983+0000","last_unstale":"2024-04-15T15:21:16.690980+0000","last_undegraded":"2024-04-15T15:21:16.690980+0000","last_fullsized":"2024-04-15T15:21:16.690980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:24.468426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732374+0000","last_change":"2024-04-15T15:21:11.649581+0000","last_active":"2024-04-15T15:21:16.732374+0000","last_peered":"2024-04-15T15:21:16.732374+0000","last_clean":"2024-04-15T15:21:16.732374+0000","last_became_active":"2024-04-15T15:21:11.649255+0000","last_became_peered":"2024-04-15T15:21:11.649255+0000","last_unstale":"2024-04-15T15:21:16.732374+0000","last_undegraded":"2024-04-15T15:21:16.732374+0000","last_fullsized":"2024-04-15T15:21:16.732374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691111+0000","last_change":"2024-04-15T15:21:11.667227+0000","last_active":"2024-04-15T15:21:16.691111+0000","last_peered":"2024-04-15T15:21:16.691111+0000","last_clean":"2024-04-15T15:21:16.691111+0000","last_became_active":"2024-04-15T15:21:11.663970+0000","last_became_peered":"2024-04-15T15:21:11.663970+0000","last_unstale":"2024-04-15T15:21:16.691111+0000","last_undegraded":"2024-04-15T15:21:16.691111+0000","last_fullsized":"2024-04-15T15:21:16.691111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:02.359534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729911+0000","last_change":"2024-04-15T15:21:11.659069+0000","last_active":"2024-04-15T15:21:16.729911+0000","last_peered":"2024-04-15T15:21:16.729911+0000","last_clean":"2024-04-15T15:21:16.729911+0000","last_became_active":"2024-04-15T15:21:11.658826+0000","last_became_peered":"2024-04-15T15:21:11.658826+0000","last_unstale":"2024-04-15T15:21:16.729911+0000","last_undegraded":"2024-04-15T15:21:16.729911+0000","last_fullsized":"2024-04-15T15:21:16.729911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:34:00.069973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730161+0000","last_change":"2024-04-15T15:21:11.662189+0000","last_active":"2024-04-15T15:21:16.730161+0000","last_peered":"2024-04-15T15:21:16.730161+0000","last_clean":"2024-04-15T15:21:16.730161+0000","last_became_active":"2024-04-15T15:21:11.661850+0000","last_became_peered":"2024-04-15T15:21:11.661850+0000","last_unstale":"2024-04-15T15:21:16.730161+0000","last_undegraded":"2024-04-15T15:21:16.730161+0000","last_fullsized":"2024-04-15T15:21:16.730161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730978+0000","last_change":"2024-04-15T15:21:11.656094+0000","last_active":"2024-04-15T15:21:16.730978+0000","last_peered":"2024-04-15T15:21:16.730978+0000","last_clean":"2024-04-15T15:21:16.730978+0000","last_became_active":"2024-04-15T15:21:11.655585+0000","last_became_peered":"2024-04-15T15:21:11.655585+0000","last_unstale":"2024-04-15T15:21:16.730978+0000","last_undegraded":"2024-04-15T15:21:16.730978+0000","last_fullsized":"2024-04-15T15:21:16.730978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:26.088255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731645+0000","last_change":"2024-04-15T15:21:11.663357+0000","last_active":"2024-04-15T15:21:16.731645+0000","last_peered":"2024-04-15T15:21:16.731645+0000","last_clean":"2024-04-15T15:21:16.731645+0000","last_became_active":"2024-04-15T15:21:11.663051+0000","last_became_peered":"2024-04-15T15:21:11.663051+0000","last_unstale":"2024-04-15T15:21:16.731645+0000","last_undegraded":"2024-04-15T15:21:16.731645+0000","last_fullsized":"2024-04-15T15:21:16.731645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:53.740250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691911+0000","last_change":"2024-04-15T15:21:11.661796+0000","last_active":"2024-04-15T15:21:16.691911+0000","last_peered":"2024-04-15T15:21:16.691911+0000","last_clean":"2024-04-15T15:21:16.691911+0000","last_became_active":"2024-04-15T15:21:11.661288+0000","last_became_peered":"2024-04-15T15:21:11.661288+0000","last_unstale":"2024-04-15T15:21:16.691911+0000","last_undegraded":"2024-04-15T15:21:16.691911+0000","last_fullsized":"2024-04-15T15:21:16.691911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:26.088255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690954+0000","last_change":"2024-04-15T15:21:11.663287+0000","last_active":"2024-04-15T15:21:16.690954+0000","last_peered":"2024-04-15T15:21:16.690954+0000","last_clean":"2024-04-15T15:21:16.690954+0000","last_became_active":"2024-04-15T15:21:11.662684+0000","last_became_peered":"2024-04-15T15:21:11.662684+0000","last_unstale":"2024-04-15T15:21:16.690954+0000","last_undegraded":"2024-04-15T15:21:16.690954+0000","last_fullsized":"2024-04-15T15:21:16.690954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729440+0000","last_change":"2024-04-15T15:21:11.657278+0000","last_active":"2024-04-15T15:21:16.729440+0000","last_peered":"2024-04-15T15:21:16.729440+0000","last_clean":"2024-04-15T15:21:16.729440+0000","last_became_active":"2024-04-15T15:21:11.656878+0000","last_became_peered":"2024-04-15T15:21:11.656878+0000","last_unstale":"2024-04-15T15:21:16.729440+0000","last_undegraded":"2024-04-15T15:21:16.729440+0000","last_fullsized":"2024-04-15T15:21:16.729440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730993+0000","last_change":"2024-04-15T15:21:11.646900+0000","last_active":"2024-04-15T15:21:16.730993+0000","last_peered":"2024-04-15T15:21:16.730993+0000","last_clean":"2024-04-15T15:21:16.730993+0000","last_became_active":"2024-04-15T15:21:11.646504+0000","last_became_peered":"2024-04-15T15:21:11.646504+0000","last_unstale":"2024-04-15T15:21:16.730993+0000","last_undegraded":"2024-04-15T15:21:16.730993+0000","last_fullsized":"2024-04-15T15:21:16.730993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732733+0000","last_change":"2024-04-15T15:21:11.666961+0000","last_active":"2024-04-15T15:21:16.732733+0000","last_peered":"2024-04-15T15:21:16.732733+0000","last_clean":"2024-04-15T15:21:16.732733+0000","last_became_active":"2024-04-15T15:21:11.666715+0000","last_became_peered":"2024-04-15T15:21:11.666715+0000","last_unstale":"2024-04-15T15:21:16.732733+0000","last_undegraded":"2024-04-15T15:21:16.732733+0000","last_fullsized":"2024-04-15T15:21:16.732733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685618+0000","last_change":"2024-04-15T15:21:11.662851+0000","last_active":"2024-04-15T15:21:16.685618+0000","last_peered":"2024-04-15T15:21:16.685618+0000","last_clean":"2024-04-15T15:21:16.685618+0000","last_became_active":"2024-04-15T15:21:11.662536+0000","last_became_peered":"2024-04-15T15:21:11.662536+0000","last_unstale":"2024-04-15T15:21:16.685618+0000","last_undegraded":"2024-04-15T15:21:16.685618+0000","last_fullsized":"2024-04-15T15:21:16.685618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:53.932651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730237+0000","last_change":"2024-04-15T15:21:11.670651+0000","last_active":"2024-04-15T15:21:16.730237+0000","last_peered":"2024-04-15T15:21:16.730237+0000","last_clean":"2024-04-15T15:21:16.730237+0000","last_became_active":"2024-04-15T15:21:11.670354+0000","last_became_peered":"2024-04-15T15:21:11.670354+0000","last_unstale":"2024-04-15T15:21:16.730237+0000","last_undegraded":"2024-04-15T15:21:16.730237+0000","last_fullsized":"2024-04-15T15:21:16.730237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730494+0000","last_change":"2024-04-15T15:21:11.670652+0000","last_active":"2024-04-15T15:21:16.730494+0000","last_peered":"2024-04-15T15:21:16.730494+0000","last_clean":"2024-04-15T15:21:16.730494+0000","last_became_active":"2024-04-15T15:21:11.670396+0000","last_became_peered":"2024-04-15T15:21:11.670396+0000","last_unstale":"2024-04-15T15:21:16.730494+0000","last_undegraded":"2024-04-15T15:21:16.730494+0000","last_fullsized":"2024-04-15T15:21:16.730494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730636+0000","last_change":"2024-04-15T15:21:11.665992+0000","last_active":"2024-04-15T15:21:16.730636+0000","last_peered":"2024-04-15T15:21:16.730636+0000","last_clean":"2024-04-15T15:21:16.730636+0000","last_became_active":"2024-04-15T15:21:11.665531+0000","last_became_peered":"2024-04-15T15:21:11.665531+0000","last_unstale":"2024-04-15T15:21:16.730636+0000","last_undegraded":"2024-04-15T15:21:16.730636+0000","last_fullsized":"2024-04-15T15:21:16.730636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:09.770327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732177+0000","last_change":"2024-04-15T15:21:11.665238+0000","last_active":"2024-04-15T15:21:16.732177+0000","last_peered":"2024-04-15T15:21:16.732177+0000","last_clean":"2024-04-15T15:21:16.732177+0000","last_became_active":"2024-04-15T15:21:11.664931+0000","last_became_peered":"2024-04-15T15:21:11.664931+0000","last_unstale":"2024-04-15T15:21:16.732177+0000","last_undegraded":"2024-04-15T15:21:16.732177+0000","last_fullsized":"2024-04-15T15:21:16.732177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731045+0000","last_change":"2024-04-15T15:21:11.655222+0000","last_active":"2024-04-15T15:21:16.731045+0000","last_peered":"2024-04-15T15:21:16.731045+0000","last_clean":"2024-04-15T15:21:16.731045+0000","last_became_active":"2024-04-15T15:21:11.654826+0000","last_became_peered":"2024-04-15T15:21:11.654826+0000","last_unstale":"2024-04-15T15:21:16.731045+0000","last_undegraded":"2024-04-15T15:21:16.731045+0000","last_fullsized":"2024-04-15T15:21:16.731045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:20.066936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690947+0000","last_change":"2024-04-15T15:21:11.662632+0000","last_active":"2024-04-15T15:21:16.690947+0000","last_peered":"2024-04-15T15:21:16.690947+0000","last_clean":"2024-04-15T15:21:16.690947+0000","last_became_active":"2024-04-15T15:21:11.661632+0000","last_became_peered":"2024-04-15T15:21:11.661632+0000","last_unstale":"2024-04-15T15:21:16.690947+0000","last_undegraded":"2024-04-15T15:21:16.690947+0000","last_fullsized":"2024-04-15T15:21:16.690947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730098+0000","last_change":"2024-04-15T15:21:11.655451+0000","last_active":"2024-04-15T15:21:16.730098+0000","last_peered":"2024-04-15T15:21:16.730098+0000","last_clean":"2024-04-15T15:21:16.730098+0000","last_became_active":"2024-04-15T15:21:11.655138+0000","last_became_peered":"2024-04-15T15:21:11.655138+0000","last_unstale":"2024-04-15T15:21:16.730098+0000","last_undegraded":"2024-04-15T15:21:16.730098+0000","last_fullsized":"2024-04-15T15:21:16.730098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691160+0000","last_change":"2024-04-15T15:21:11.662669+0000","last_active":"2024-04-15T15:21:16.691160+0000","last_peered":"2024-04-15T15:21:16.691160+0000","last_clean":"2024-04-15T15:21:16.691160+0000","last_became_active":"2024-04-15T15:21:11.662385+0000","last_became_peered":"2024-04-15T15:21:11.662385+0000","last_unstale":"2024-04-15T15:21:16.691160+0000","last_undegraded":"2024-04-15T15:21:16.691160+0000","last_fullsized":"2024-04-15T15:21:16.691160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:55.392737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731209+0000","last_change":"2024-04-15T15:21:11.640925+0000","last_active":"2024-04-15T15:21:16.731209+0000","last_peered":"2024-04-15T15:21:16.731209+0000","last_clean":"2024-04-15T15:21:16.731209+0000","last_became_active":"2024-04-15T15:21:11.640618+0000","last_became_peered":"2024-04-15T15:21:11.640618+0000","last_unstale":"2024-04-15T15:21:16.731209+0000","last_undegraded":"2024-04-15T15:21:16.731209+0000","last_fullsized":"2024-04-15T15:21:16.731209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730901+0000","last_change":"2024-04-15T15:21:11.653243+0000","last_active":"2024-04-15T15:21:16.730901+0000","last_peered":"2024-04-15T15:21:16.730901+0000","last_clean":"2024-04-15T15:21:16.730901+0000","last_became_active":"2024-04-15T15:21:11.652962+0000","last_became_peered":"2024-04-15T15:21:11.652962+0000","last_unstale":"2024-04-15T15:21:16.730901+0000","last_undegraded":"2024-04-15T15:21:16.730901+0000","last_fullsized":"2024-04-15T15:21:16.730901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730496+0000","last_change":"2024-04-15T15:21:11.662774+0000","last_active":"2024-04-15T15:21:16.730496+0000","last_peered":"2024-04-15T15:21:16.730496+0000","last_clean":"2024-04-15T15:21:16.730496+0000","last_became_active":"2024-04-15T15:21:11.662488+0000","last_became_peered":"2024-04-15T15:21:11.662488+0000","last_unstale":"2024-04-15T15:21:16.730496+0000","last_undegraded":"2024-04-15T15:21:16.730496+0000","last_fullsized":"2024-04-15T15:21:16.730496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:09.923283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:21:35.138 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-15T15:21:35.139 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T15:21:35.139 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-15T15:21:35.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-15T15:21:35.139 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-15T15:21:35.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.277+0000 7f612e47b640 1 Processor -- start 2024-04-15T15:21:35.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.277+0000 7f612e47b640 1 -- start start 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612e47b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f61280d3480 0x7f61280b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612e47b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f61280b6790 0x7f61280b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612e47b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61280cff70 con 0x7f612815ea40 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612e47b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f61280d0250 con 0x7f61280b6790 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612e47b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f61280d0530 con 0x7f61280d3480 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612ca18640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f61280b6790 0x7f61280b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f6127fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f612815ea40 0x7f61280b5b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56264/0 (socket says 172.21.15.12:56264) 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f6127fff640 1 -- 172.21.15.12:0/2797506077 learned_addr learned my addr 172.21.15.12:0/2797506077 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f611ffff640 1 --2- 172.21.15.12:0/2797506077 >> v2:172.21.15.55:3300/0 conn(0x7f61280d3480 0x7f61280b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612ca18640 1 -- 172.21.15.12:0/2797506077 >> v2:172.21.15.55:3300/0 conn(0x7f61280d3480 msgr2=0x7f61280b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:35.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612ca18640 1 --2- 172.21.15.12:0/2797506077 >> v2:172.21.15.55:3300/0 conn(0x7f61280d3480 0x7f61280b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612ca18640 1 -- 172.21.15.12:0/2797506077 >> v1:172.21.15.12:6789/0 conn(0x7f612815ea40 legacy=0x7f61280b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f612ca18640 1 -- 172.21.15.12:0/2797506077 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61280d0810 con 0x7f61280b6790 2024-04-15T15:21:35.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.278+0000 7f61277fe640 1 -- 172.21.15.12:0/2797506077 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356940166 0 0) 0x7f61280cff70 con 0x7f612815ea40 2024-04-15T15:21:35.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.279+0000 7f612ca18640 1 --2- 172.21.15.12:0/2797506077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61280b6790 0x7f61280b3e80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f612808cc60 tx=0x7f612002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fd961a2d1d0498 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.279+0000 7f61277fe640 1 -- 172.21.15.12:0/2797506077 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120039070 con 0x7f61280b6790 2024-04-15T15:21:35.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.279+0000 7f61277fe640 1 -- 172.21.15.12:0/2797506077 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f612002cd00 con 0x7f61280b6790 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.279+0000 7f61277fe640 1 -- 172.21.15.12:0/2797506077 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120034660 con 0x7f61280b6790 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.280+0000 7f612e47b640 1 -- 172.21.15.12:0/2797506077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61280b6790 msgr2=0x7f61280b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.280+0000 7f612e47b640 1 --2- 172.21.15.12:0/2797506077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61280b6790 0x7f61280b3e80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f612808cc60 tx=0x7f612002bf50 comp rx=0 tx=0).stop 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.280+0000 7f612e47b640 1 -- 172.21.15.12:0/2797506077 shutdown_connections 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.280+0000 7f612e47b640 1 --2- 172.21.15.12:0/2797506077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61280b6790 0x7f61280b3e80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.280+0000 7f612e47b640 1 --2- 172.21.15.12:0/2797506077 >> v2:172.21.15.55:3300/0 conn(0x7f61280d3480 0x7f61280b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.280+0000 7f612e47b640 1 -- 172.21.15.12:0/2797506077 >> 172.21.15.12:0/2797506077 conn(0x7f612808bf60 msgr2=0x7f61280d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.280+0000 7f612e47b640 1 -- 172.21.15.12:0/2797506077 shutdown_connections 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.280+0000 7f612e47b640 1 -- 172.21.15.12:0/2797506077 wait complete. 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.281+0000 7f612e47b640 1 Processor -- start 2024-04-15T15:21:35.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.281+0000 7f612e47b640 1 -- start start 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612e47b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61280b6790 0x7f61280ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612e47b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f61280d3480 0x7f6128160740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612e47b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f612815ea40 0x7f6128156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612e47b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f61280b3a40 con 0x7f612815ea40 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612e47b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f61280b38c0 con 0x7f61280b6790 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612e47b640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f612815ee20 con 0x7f61280d3480 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f612815ea40 0x7f6128156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f612815ea40 0x7f6128156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46606/0 (socket says 172.21.15.12:46606) 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 -- 172.21.15.12:0/1197596842 learned_addr learned my addr 172.21.15.12:0/1197596842 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f6127fff640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61280b6790 0x7f61280ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f611ffff640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f61280d3480 0x7f6128160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 -- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f61280d3480 msgr2=0x7f6128160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f61280d3480 0x7f6128160740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 -- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61280b6790 msgr2=0x7f61280ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61280b6790 0x7f61280ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 -- 172.21.15.12:0/1197596842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6128162090 con 0x7f612815ea40 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f6127fff640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61280b6790 0x7f61280ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.282+0000 7f612ca18640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f612815ea40 0x7f6128156e50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6120009e80 tx=0x7f6120004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.283+0000 7f61257fa640 1 -- 172.21.15.12:0/1197596842 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120039050 con 0x7f612815ea40 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.283+0000 7f61257fa640 1 -- 172.21.15.12:0/1197596842 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6120045070 con 0x7f612815ea40 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.283+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6128161db0 con 0x7f612815ea40 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.283+0000 7f61257fa640 1 -- 172.21.15.12:0/1197596842 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120009450 con 0x7f612815ea40 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.283+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612815ff40 con 0x7f612815ea40 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.284+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61280689e0 con 0x7f612815ea40 2024-04-15T15:21:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.284+0000 7f61257fa640 1 -- 172.21.15.12:0/1197596842 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f612003e850 con 0x7f612815ea40 2024-04-15T15:21:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.284+0000 7f61257fa640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f610806c8f0 0x7f610806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.285+0000 7f61257fa640 1 -- 172.21.15.12:0/1197596842 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f61200aeb20 con 0x7f612815ea40 2024-04-15T15:21:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.285+0000 7f6127fff640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f610806c8f0 0x7f610806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.285+0000 7f6127fff640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f610806c8f0 0x7f610806edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6118000f60 tx=0x7f6118007cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.289+0000 7f61257fa640 1 -- 172.21.15.12:0/1197596842 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f612007cdc0 con 0x7f612815ea40 2024-04-15T15:21:35.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.527+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f61280d1cd0 con 0x7f612815ea40 2024-04-15T15:21:35.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.530+0000 7f61257fa640 1 -- 172.21.15.12:0/1197596842 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174133 (secure 0 0 0) 0x7f612007c6c0 con 0x7f612815ea40 2024-04-15T15:21:35.533 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.532+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f610806c8f0 msgr2=0x7f610806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.532+0000 7f612e47b640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f610806c8f0 0x7f610806edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6118000f60 tx=0x7f6118007cc0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.532+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f612815ea40 msgr2=0x7f6128156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.532+0000 7f612e47b640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f612815ea40 0x7f6128156e50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6120009e80 tx=0x7f6120004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.533+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 shutdown_connections 2024-04-15T15:21:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.533+0000 7f612e47b640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f612815ea40 0x7f6128156e50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.533+0000 7f612e47b640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f61280d3480 0x7f6128160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.533+0000 7f612e47b640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f610806c8f0 0x7f610806edb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.533+0000 7f612e47b640 1 --2- 172.21.15.12:0/1197596842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61280b6790 0x7f61280ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.533+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 >> 172.21.15.12:0/1197596842 conn(0x7f612808bf60 msgr2=0x7f61280b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.533+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 shutdown_connections 2024-04-15T15:21:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.533+0000 7f612e47b640 1 -- 172.21.15.12:0/1197596842 wait complete. 2024-04-15T15:21:35.545 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":746784251},{"type":"v1","addr":"172.21.15.55:6833","nonce":746784251}]},"active_addr":"172.21.15.55:6833/746784251","active_change":"2024-04-15T15:20:50.579398+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2272823330}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":940533249}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2611451955}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":1906848872}]}]} 2024-04-15T15:21:35.548 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-15T15:21:35.548 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-15T15:21:35.548 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.652+0000 7f8375074640 1 Processor -- start 2024-04-15T15:21:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.652+0000 7f8375074640 1 -- start start 2024-04-15T15:21:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.653+0000 7f8375074640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f837015e700 0x7f83700b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.653+0000 7f8375074640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f83700d2360 0x7f83700d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.653+0000 7f8375074640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83700b47a0 con 0x7f837015eae0 2024-04-15T15:21:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.653+0000 7f8375074640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f83700b53b0 con 0x7f837015e700 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.653+0000 7f8375074640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f83700b4a80 con 0x7f83700d2360 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.653+0000 7f836ed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f837015e700 0x7f83700b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.653+0000 7f836ed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f837015e700 0x7f83700b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46608/0 (socket says 172.21.15.12:46608) 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.653+0000 7f836ed9d640 1 -- 172.21.15.12:0/2700639810 learned_addr learned my addr 172.21.15.12:0/2700639810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.654+0000 7f836659c640 1 --2- 172.21.15.12:0/2700639810 >> v2:172.21.15.55:3300/0 conn(0x7f83700d2360 0x7f83700d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.654+0000 7f836ed9d640 1 -- 172.21.15.12:0/2700639810 >> v2:172.21.15.55:3300/0 conn(0x7f83700d2360 msgr2=0x7f83700d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.654+0000 7f836ed9d640 1 --2- 172.21.15.12:0/2700639810 >> v2:172.21.15.55:3300/0 conn(0x7f83700d2360 0x7f83700d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.654+0000 7f836ed9d640 1 -- 172.21.15.12:0/2700639810 >> v1:172.21.15.55:6789/0 conn(0x7f837015eae0 legacy=0x7f83700cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.654+0000 7f836ed9d640 1 -- 172.21.15.12:0/2700639810 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83700b4d60 con 0x7f837015e700 2024-04-15T15:21:35.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f836ed9d640 1 --2- 172.21.15.12:0/2700639810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b5840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f837008c7b0 tx=0x7f836002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e4cd56a137e8b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f836e59c640 1 -- 172.21.15.12:0/2700639810 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360004230 con 0x7f837015e700 2024-04-15T15:21:35.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f836e59c640 1 -- 172.21.15.12:0/2700639810 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83600043d0 con 0x7f837015e700 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f836e59c640 1 -- 172.21.15.12:0/2700639810 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360030840 con 0x7f837015e700 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f8375074640 1 -- 172.21.15.12:0/2700639810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 msgr2=0x7f83700b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f8375074640 1 --2- 172.21.15.12:0/2700639810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b5840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f837008c7b0 tx=0x7f836002b980 comp rx=0 tx=0).stop 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f8375074640 1 -- 172.21.15.12:0/2700639810 shutdown_connections 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f8375074640 1 --2- 172.21.15.12:0/2700639810 >> v2:172.21.15.55:3300/0 conn(0x7f83700d2360 0x7f83700d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f8375074640 1 --2- 172.21.15.12:0/2700639810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b5840 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f8375074640 1 -- 172.21.15.12:0/2700639810 >> 172.21.15.12:0/2700639810 conn(0x7f837008bf60 msgr2=0x7f83700d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f8375074640 1 -- 172.21.15.12:0/2700639810 shutdown_connections 2024-04-15T15:21:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.655+0000 7f8375074640 1 -- 172.21.15.12:0/2700639810 wait complete. 2024-04-15T15:21:35.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.657+0000 7f8375074640 1 Processor -- start 2024-04-15T15:21:35.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.657+0000 7f8375074640 1 -- start start 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f8375074640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83700d2360 0x7f83700ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f8375074640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f8375074640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f837015eae0 0x7f83700b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f8375074640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f83700b5d80 con 0x7f837015e700 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f8375074640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f83700b4450 con 0x7f837015eae0 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f8375074640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f83700d1d40 con 0x7f83700d2360 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46624/0 (socket says 172.21.15.12:46624) 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 -- 172.21.15.12:0/1075819207 learned_addr learned my addr 172.21.15.12:0/1075819207 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836ed9d640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83700d2360 0x7f83700ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836f59e640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f837015eae0 0x7f83700b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 -- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83700d2360 msgr2=0x7f83700ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83700d2360 0x7f83700ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 -- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f837015eae0 msgr2=0x7f83700b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f837015eae0 0x7f83700b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 -- 172.21.15.12:0/1075819207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83700b95f0 con 0x7f837015e700 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f836659c640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b37e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f835800e9b0 tx=0x7f835800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.658+0000 7f8367fff640 1 -- 172.21.15.12:0/1075819207 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8358017070 con 0x7f837015e700 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.659+0000 7f8367fff640 1 -- 172.21.15.12:0/1075819207 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f835800c9a0 con 0x7f837015e700 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.659+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83700b98d0 con 0x7f837015e700 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.659+0000 7f8367fff640 1 -- 172.21.15.12:0/1075819207 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835800cd90 con 0x7f837015e700 2024-04-15T15:21:35.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.659+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83700b9e90 con 0x7f837015e700 2024-04-15T15:21:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.660+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83700689e0 con 0x7f837015e700 2024-04-15T15:21:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.660+0000 7f8367fff640 1 -- 172.21.15.12:0/1075819207 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f835801a070 con 0x7f837015e700 2024-04-15T15:21:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.660+0000 7f8367fff640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f835006c920 0x7f835006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.661+0000 7f8367fff640 1 -- 172.21.15.12:0/1075819207 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f835808c170 con 0x7f837015e700 2024-04-15T15:21:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.661+0000 7f836ed9d640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f835006c920 0x7f835006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.661+0000 7f836ed9d640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f835006c920 0x7f835006ede0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8360035000 tx=0x7f8360008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.665+0000 7f8367fff640 1 -- 172.21.15.12:0/1075819207 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8358059c10 con 0x7f837015e700 2024-04-15T15:21:35.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.854+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83700ba500 con 0x7f837015e700 2024-04-15T15:21:35.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.856+0000 7f8367fff640 1 -- 172.21.15.12:0/1075819207 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f835805df10 con 0x7f837015e700 2024-04-15T15:21:35.857 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:35.857 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:35.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f835006c920 msgr2=0x7f835006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f835006c920 0x7f835006ede0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8360035000 tx=0x7f8360008030 comp rx=0 tx=0).stop 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 msgr2=0x7f83700b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b37e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f835800e9b0 tx=0x7f835800ee70 comp rx=0 tx=0).stop 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 shutdown_connections 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f835006c920 0x7f835006ede0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f837015eae0 0x7f83700b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f837015e700 0x7f83700b37e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 --2- 172.21.15.12:0/1075819207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83700d2360 0x7f83700ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 >> 172.21.15.12:0/1075819207 conn(0x7f837008bf60 msgr2=0x7f83700cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 shutdown_connections 2024-04-15T15:21:35.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.858+0000 7f8375074640 1 -- 172.21.15.12:0/1075819207 wait complete. 2024-04-15T15:21:35.871 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-15T15:21:35.871 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.973+0000 7f60950ce640 1 Processor -- start 2024-04-15T15:21:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.973+0000 7f60950ce640 1 -- start start 2024-04-15T15:21:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f60950ce640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f609015c8d0 0x7f60900cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f60950ce640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f609009b650 0x7f609009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f60950ce640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60900d0350 con 0x7f609009af10 2024-04-15T15:21:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f60950ce640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f60900b5b00 con 0x7f609009b650 2024-04-15T15:21:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f60950ce640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f609009b340 con 0x7f609015c8d0 2024-04-15T15:21:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f6087fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f609009b650 0x7f609009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f6087fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f609009b650 0x7f609009a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46636/0 (socket says 172.21.15.12:46636) 2024-04-15T15:21:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f6087fff640 1 -- 172.21.15.12:0/1595188550 learned_addr learned my addr 172.21.15.12:0/1595188550 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.974+0000 7f608ed9d640 1 --2- 172.21.15.12:0/1595188550 >> v2:172.21.15.55:3300/0 conn(0x7f609015c8d0 0x7f60900cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.975+0000 7f6087fff640 1 -- 172.21.15.12:0/1595188550 >> v2:172.21.15.55:3300/0 conn(0x7f609015c8d0 msgr2=0x7f60900cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.975+0000 7f6087fff640 1 --2- 172.21.15.12:0/1595188550 >> v2:172.21.15.55:3300/0 conn(0x7f609015c8d0 0x7f60900cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.975+0000 7f6087fff640 1 -- 172.21.15.12:0/1595188550 >> v1:172.21.15.55:6789/0 conn(0x7f609009af10 legacy=0x7f60900d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.975+0000 7f6087fff640 1 -- 172.21.15.12:0/1595188550 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60900ce100 con 0x7f609009b650 2024-04-15T15:21:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f6087fff640 1 --2- 172.21.15.12:0/1595188550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f609009a9d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f609009b620 tx=0x7f607802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf8c2c6f59796b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f608e59c640 1 -- 172.21.15.12:0/1595188550 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078035020 con 0x7f609009b650 2024-04-15T15:21:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f608e59c640 1 -- 172.21.15.12:0/1595188550 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6078031870 con 0x7f609009b650 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f608e59c640 1 -- 172.21.15.12:0/1595188550 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078031c40 con 0x7f609009b650 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f60950ce640 1 -- 172.21.15.12:0/1595188550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 msgr2=0x7f609009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f60950ce640 1 --2- 172.21.15.12:0/1595188550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f609009a9d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f609009b620 tx=0x7f607802b910 comp rx=0 tx=0).stop 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f60950ce640 1 -- 172.21.15.12:0/1595188550 shutdown_connections 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f60950ce640 1 --2- 172.21.15.12:0/1595188550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f609009a9d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f60950ce640 1 --2- 172.21.15.12:0/1595188550 >> v2:172.21.15.55:3300/0 conn(0x7f609015c8d0 0x7f60900cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f60950ce640 1 -- 172.21.15.12:0/1595188550 >> 172.21.15.12:0/1595188550 conn(0x7f60900b3fb0 msgr2=0x7f60900b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f60950ce640 1 -- 172.21.15.12:0/1595188550 shutdown_connections 2024-04-15T15:21:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.976+0000 7f60950ce640 1 -- 172.21.15.12:0/1595188550 wait complete. 2024-04-15T15:21:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f60950ce640 1 Processor -- start 2024-04-15T15:21:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f60950ce640 1 -- start start 2024-04-15T15:21:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f60950ce640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f609009af10 0x7f60900cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f60950ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f60900b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f60950ce640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60900b48d0 0x7f609009ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f60950ce640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f60900b6fa0 con 0x7f609009b650 2024-04-15T15:21:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f60950ce640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f60900b5790 con 0x7f60900b48d0 2024-04-15T15:21:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f60950ce640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f60900b7120 con 0x7f609009af10 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f6087fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f60900b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.978+0000 7f6087fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f60900b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46642/0 (socket says 172.21.15.12:46642) 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f6087fff640 1 -- 172.21.15.12:0/4284961357 learned_addr learned my addr 172.21.15.12:0/4284961357 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f608f59e640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60900b48d0 0x7f609009ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f608ed9d640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f609009af10 0x7f60900cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f6087fff640 1 -- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f609009af10 msgr2=0x7f60900cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f6087fff640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f609009af10 0x7f60900cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f6087fff640 1 -- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60900b48d0 msgr2=0x7f609009ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f6087fff640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60900b48d0 0x7f609009ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f6087fff640 1 -- 172.21.15.12:0/4284961357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609009fa00 con 0x7f609009b650 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f6087fff640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f60900b4390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f607802b950 tx=0x7f60780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f60877fe640 1 -- 172.21.15.12:0/4284961357 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078040070 con 0x7f609009b650 2024-04-15T15:21:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f60877fe640 1 -- 172.21.15.12:0/4284961357 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6078030d00 con 0x7f609009b650 2024-04-15T15:21:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609009fce0 con 0x7f609009b650 2024-04-15T15:21:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f60877fe640 1 -- 172.21.15.12:0/4284961357 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078035040 con 0x7f609009b650 2024-04-15T15:21:35.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.979+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60900cc840 con 0x7f609009b650 2024-04-15T15:21:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.980+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60900b4ce0 con 0x7f609009b650 2024-04-15T15:21:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.981+0000 7f60877fe640 1 -- 172.21.15.12:0/4284961357 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f607804b060 con 0x7f609009b650 2024-04-15T15:21:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.981+0000 7f60877fe640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f607006c920 0x7f607006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.981+0000 7f60877fe640 1 -- 172.21.15.12:0/4284961357 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f60780adb00 con 0x7f609009b650 2024-04-15T15:21:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.981+0000 7f608ed9d640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f607006c920 0x7f607006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:35.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.982+0000 7f608ed9d640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f607006c920 0x7f607006ede0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6080009b40 tx=0x7f6080009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:35.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:35.986+0000 7f60877fe640 1 -- 172.21.15.12:0/4284961357 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60900b4ce0 con 0x7f609009b650 2024-04-15T15:21:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.172+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60900690f0 con 0x7f609009b650 2024-04-15T15:21:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.173+0000 7f60877fe640 1 -- 172.21.15.12:0/4284961357 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f607807b520 con 0x7f609009b650 2024-04-15T15:21:36.175 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:36.175 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:36.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f607006c920 msgr2=0x7f607006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f607006c920 0x7f607006ede0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6080009b40 tx=0x7f6080009190 comp rx=0 tx=0).stop 2024-04-15T15:21:36.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 msgr2=0x7f60900b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f60900b4390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f607802b950 tx=0x7f60780046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 shutdown_connections 2024-04-15T15:21:36.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f607006c920 0x7f607006ede0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60900b48d0 0x7f609009ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f609009b650 0x7f60900b4390 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 --2- 172.21.15.12:0/4284961357 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f609009af10 0x7f60900cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.175+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 >> 172.21.15.12:0/4284961357 conn(0x7f60900b3fb0 msgr2=0x7f60900c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.176+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 shutdown_connections 2024-04-15T15:21:36.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.176+0000 7f60950ce640 1 -- 172.21.15.12:0/4284961357 wait complete. 2024-04-15T15:21:36.189 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-15T15:21:36.189 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-15T15:21:36.189 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-15T15:21:36.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-15T15:21:36.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-15T15:21:36.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-15T15:21:36.191 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-15T15:21:36.191 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-15T15:21:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.367+0000 7fc5f8869640 1 Processor -- start 2024-04-15T15:21:36.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.367+0000 7fc5f8869640 1 -- start start 2024-04-15T15:21:36.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.367+0000 7f52ec5f4640 1 Processor -- start 2024-04-15T15:21:36.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.367+0000 7f52ec5f4640 1 -- start start 2024-04-15T15:21:36.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.368+0000 7f52ec5f4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f52e406fe00 0x7f52e40b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f52ec5f4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 0x7f52e40d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f52ec5f4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e40b3d30 con 0x7f52e40af7f0 2024-04-15T15:21:36.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f52ec5f4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f52e40b4660 con 0x7f52e406fe00 2024-04-15T15:21:36.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f52ec5f4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f52e40b4010 con 0x7f52e415bf10 2024-04-15T15:21:36.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7fc5f8869640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc5f415a920 0x7fc5f40b3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7fc5f8869640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f40b40e0 con 0x7fc5f40b3c00 2024-04-15T15:21:36.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7fc5f8869640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f40b0f00 con 0x7fc5f40b29c0 2024-04-15T15:21:36.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7fc5f8869640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f40b0cb0 con 0x7fc5f415a920 2024-04-15T15:21:36.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f52e9b8f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f52e40af7f0 0x7f52e415b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56270/0 (socket says 172.21.15.12:56270) 2024-04-15T15:21:36.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f52e9b8f640 1 -- 172.21.15.12:0/890108721 learned_addr learned my addr 172.21.15.12:0/890108721 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:36.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f52eab91640 1 --2- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 0x7f52e40d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f52ea390640 1 --2- 172.21.15.12:0/890108721 >> v2:172.21.15.55:3300/0 conn(0x7f52e406fe00 0x7f52e40b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.369+0000 7f6dafc63640 1 Processor -- start 2024-04-15T15:21:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f8ef52b2640 1 Processor -- start 2024-04-15T15:21:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f81f52c0640 1 Processor -- start 2024-04-15T15:21:36.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f81f52c0640 1 -- start start 2024-04-15T15:21:36.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f6dafc63640 1 -- start start 2024-04-15T15:21:36.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f3eeffff640 1 Processor -- start 2024-04-15T15:21:36.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7fc16759e640 1 Processor -- start 2024-04-15T15:21:36.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f52e938e640 1 -- 172.21.15.12:0/890108721 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3211372 0 0) 0x7f52e40b3d30 con 0x7f52e40af7f0 2024-04-15T15:21:36.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f52e938e640 1 -- 172.21.15.12:0/890108721 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52d0003520 con 0x7f52e40af7f0 2024-04-15T15:21:36.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7fc5f2d9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc5f40b29c0 0x7fc5f40ad1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:58774/0 (socket says 172.21.15.12:58774) 2024-04-15T15:21:36.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7fc5f2d9d640 1 -- 172.21.15.12:0/424519197 learned_addr learned my addr 172.21.15.12:0/424519197 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:36.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7fc5f259c640 1 --2- 172.21.15.12:0/424519197 >> v2:172.21.15.12:3300/0 conn(0x7fc5f415a920 0x7fc5f40b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f3eeffff640 1 -- start start 2024-04-15T15:21:36.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7fc16759e640 1 -- start start 2024-04-15T15:21:36.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f8ef52b2640 1 -- start start 2024-04-15T15:21:36.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f1bb0ae4640 1 Processor -- start 2024-04-15T15:21:36.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f81f52c0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81e80df220 0x7f81e8038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f81f52c0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e80544c0 con 0x7f81e8053c70 2024-04-15T15:21:36.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f81f52c0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e80541e0 con 0x7f81e80501a0 2024-04-15T15:21:36.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f81f52c0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f81e8005f80 con 0x7f81e80df220 2024-04-15T15:21:36.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f6dafc63640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6da80b3470 0x7f6da80d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f6dafc63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da815bf10 0x7f6da80b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f52ea390640 1 -- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 msgr2=0x7f52e40d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f52ea390640 1 --2- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 0x7f52e40d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f52ea390640 1 -- 172.21.15.12:0/890108721 >> v1:172.21.15.12:6789/0 conn(0x7f52e40af7f0 legacy=0x7f52e415b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.370+0000 7f52ea390640 1 -- 172.21.15.12:0/890108721 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e40b42f0 con 0x7f52e406fe00 2024-04-15T15:21:36.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f52eab91640 1 --2- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 0x7f52e40d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:36.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7fc5f159a640 1 -- 172.21.15.12:0/424519197 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2343886995 0 0) 0x7fc5f40b0f00 con 0x7fc5f40b29c0 2024-04-15T15:21:36.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc5f159a640 1 -- 172.21.15.12:0/424519197 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5dc003520 con 0x7fc5f40b29c0 2024-04-15T15:21:36.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f6dafc63640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6da80d10e0 con 0x7f6da80d07c0 2024-04-15T15:21:36.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f6dafc63640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6da80b4660 con 0x7f6da80b3470 2024-04-15T15:21:36.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f6dae200640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da815bf10 0x7f6da80b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f3eeffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3ee8005ed0 0x7f3ee8006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f6dae200640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da815bf10 0x7f6da80b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33710/0 (socket says 172.21.15.12:33710) 2024-04-15T15:21:36.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f6dae200640 1 -- 172.21.15.12:0/3043866186 learned_addr learned my addr 172.21.15.12:0/3043866186 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f6dad9ff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6da80d07c0 0x7f6da815b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:58802/0 (socket says 172.21.15.12:58802) 2024-04-15T15:21:36.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f6dafc63640 1 -- 172.21.15.12:0/3043866186 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6da80b3a50 con 0x7f6da815bf10 2024-04-15T15:21:36.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f6dad1fe640 1 --2- 172.21.15.12:0/3043866186 >> v2:172.21.15.12:3300/0 conn(0x7f6da80b3470 0x7f6da80d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f3eeffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3ee8006850 0x7f3ee8006cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f3eeffff640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ee8053710 con 0x7f3ee80eacf0 2024-04-15T15:21:36.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f3eeffff640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3ee8054320 con 0x7f3ee8005ed0 2024-04-15T15:21:36.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f3eeffff640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3ee80539f0 con 0x7f3ee8006850 2024-04-15T15:21:36.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f81effff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81e80df220 0x7f81e8038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f3eeeffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3ee80eacf0 0x7f3ee804fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56302/0 (socket says 172.21.15.12:56302) 2024-04-15T15:21:36.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f3eeeffd640 1 -- 172.21.15.12:0/2504694541 learned_addr learned my addr 172.21.15.12:0/2504694541 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:36.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f3eef7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3ee8006850 0x7f3ee8006cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f81effff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f81e80df220 0x7f81e8038080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46666/0 (socket says 172.21.15.12:46666) 2024-04-15T15:21:36.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f81effff640 1 -- 172.21.15.12:0/3900808626 learned_addr learned my addr 172.21.15.12:0/3900808626 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f8ef52b2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 0x7f8ef00b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc16759e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 0x7fc16004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc16759e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc160053720 con 0x7fc160006ef0 2024-04-15T15:21:36.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc16759e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc160053a00 con 0x7fc16004ffd0 2024-04-15T15:21:36.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f8ef52b2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8ef00d1a10 0x7f8ef006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f8ef52b2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ef015ba70 con 0x7f8ef00d08a0 2024-04-15T15:21:36.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f8ef52b2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8ef00ce5b0 con 0x7f8ef00d1a10 2024-04-15T15:21:36.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f8ef52b2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef00ce890 con 0x7f8ef015b690 2024-04-15T15:21:36.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc16759e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc160054330 con 0x7fc1600ead00 2024-04-15T15:21:36.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f8eee7fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8ef00d08a0 0x7f8ef00d1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56286/0 (socket says 172.21.15.12:56286) 2024-04-15T15:21:36.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.371+0000 7f1bb0ae4640 1 -- start start 2024-04-15T15:21:36.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f1bb0ae4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1bac0cf210 0x7f1bac0d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f1bb0ae4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 0x7f1bac0d30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f8eee7fc640 1 -- 172.21.15.12:0/544528484 learned_addr learned my addr 172.21.15.12:0/544528484 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f8eef7fe640 1 --2- 172.21.15.12:0/544528484 >> v2:172.21.15.55:3300/0 conn(0x7f8ef00d1a10 0x7f8ef006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f8eeeffd640 1 --2- 172.21.15.12:0/544528484 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 0x7f8ef00b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f1bb0ae4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bac0d1e60 con 0x7f1bac15e9a0 2024-04-15T15:21:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1bb0ae4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1bac0d15d0 con 0x7f1bac0cf210 2024-04-15T15:21:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1bb0ae4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1bac0d18a0 con 0x7f1bac0cf690 2024-04-15T15:21:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc16659c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 0x7fc16004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7fc16659c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 0x7fc16004fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33734/0 (socket says 172.21.15.12:33734) 2024-04-15T15:21:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7fc16659c640 1 -- 172.21.15.12:0/421725510 learned_addr learned my addr 172.21.15.12:0/421725510 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f81effff640 1 -- 172.21.15.12:0/3900808626 >> v1:172.21.15.55:6789/0 conn(0x7f81e80501a0 legacy=0x7f81e8006a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f81effff640 1 -- 172.21.15.12:0/3900808626 >> v1:172.21.15.12:6789/0 conn(0x7f81e8053c70 legacy=0x7f81e804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f81effff640 1 -- 172.21.15.12:0/3900808626 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e8006260 con 0x7f81e80df220 2024-04-15T15:21:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1ba9d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1bac0cf210 0x7f1bac0d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1ba9d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1bac0cf210 0x7f1bac0d2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:37558/0 (socket says 172.21.15.12:37558) 2024-04-15T15:21:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1ba9d9b640 1 -- 172.21.15.12:0/1116176006 learned_addr learned my addr 172.21.15.12:0/1116176006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f3eef7fe640 1 -- 172.21.15.12:0/2504694541 >> v2:172.21.15.12:3300/0 conn(0x7f3ee8005ed0 msgr2=0x7f3ee8006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f3eef7fe640 1 --2- 172.21.15.12:0/2504694541 >> v2:172.21.15.12:3300/0 conn(0x7f3ee8005ed0 0x7f3ee8006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f3eef7fe640 1 -- 172.21.15.12:0/2504694541 >> v1:172.21.15.12:6789/0 conn(0x7f3ee80eacf0 legacy=0x7f3ee804fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f3eef7fe640 1 -- 172.21.15.12:0/2504694541 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee8053cd0 con 0x7f3ee8006850 2024-04-15T15:21:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f6dac9fd640 1 -- 172.21.15.12:0/3043866186 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 463941354 0 0) 0x7f6da80d10e0 con 0x7f6da80d07c0 2024-04-15T15:21:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f6dac9fd640 1 -- 172.21.15.12:0/3043866186 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d90003520 con 0x7f6da80d07c0 2024-04-15T15:21:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f6dae200640 1 -- 172.21.15.12:0/3043866186 >> v2:172.21.15.12:3300/0 conn(0x7f6da80b3470 msgr2=0x7f6da80d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f6dae200640 1 --2- 172.21.15.12:0/3043866186 >> v2:172.21.15.12:3300/0 conn(0x7f6da80b3470 0x7f6da80d0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f6dae200640 1 -- 172.21.15.12:0/3043866186 >> v1:172.21.15.55:6789/0 conn(0x7f6da80d07c0 legacy=0x7f6da815b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f6dae200640 1 -- 172.21.15.12:0/3043866186 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da80b3d30 con 0x7f6da815bf10 2024-04-15T15:21:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1baad9d640 1 --2- 172.21.15.12:0/1116176006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 0x7f1bac0d30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc5f259c640 1 -- 172.21.15.12:0/424519197 >> v1:172.21.15.55:6789/0 conn(0x7fc5f40b29c0 legacy=0x7fc5f40ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc5f259c640 1 -- 172.21.15.12:0/424519197 >> v1:172.21.15.12:6789/0 conn(0x7fc5f40b3c00 legacy=0x7fc5f40b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7fc5f259c640 1 -- 172.21.15.12:0/424519197 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f40b11e0 con 0x7fc5f415a920 2024-04-15T15:21:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7fc16659c640 1 -- 172.21.15.12:0/421725510 >> v1:172.21.15.55:6789/0 conn(0x7fc16004ffd0 legacy=0x7fc160037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1ba9d9b640 1 -- 172.21.15.12:0/1116176006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 msgr2=0x7f1bac0d30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1ba9d9b640 1 --2- 172.21.15.12:0/1116176006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 0x7f1bac0d30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7fc5f259c640 1 --2- 172.21.15.12:0/424519197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5f415a920 0x7fc5f40b3160 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40b0e30 tx=0x7fc5e402ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f715faca2bb3ed9f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1ba9d9b640 1 -- 172.21.15.12:0/1116176006 >> v1:172.21.15.55:6789/0 conn(0x7f1bac15e9a0 legacy=0x7f1bac0d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f1ba9d9b640 1 -- 172.21.15.12:0/1116176006 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bac0d1b80 con 0x7f1bac0cf210 2024-04-15T15:21:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.372+0000 7f52ea390640 1 --2- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52e406fe00 0x7f52e40b3470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009d90 tx=0x7f52dc02bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cac1ac54372e2cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7f52e938e640 1 -- 172.21.15.12:0/890108721 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc039070 con 0x7f52e406fe00 2024-04-15T15:21:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f52e938e640 1 -- 172.21.15.12:0/890108721 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc02cc70 con 0x7f52e406fe00 2024-04-15T15:21:36.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7fc16659c640 1 -- 172.21.15.12:0/421725510 >> v1:172.21.15.12:6789/0 conn(0x7fc160006ef0 legacy=0x7fc1600382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f8eef7fe640 1 -- 172.21.15.12:0/544528484 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 msgr2=0x7f8ef00b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f8eef7fe640 1 --2- 172.21.15.12:0/544528484 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 0x7f8ef00b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f8eef7fe640 1 -- 172.21.15.12:0/544528484 >> v1:172.21.15.12:6789/0 conn(0x7f8ef00d08a0 legacy=0x7f8ef00d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f8eef7fe640 1 -- 172.21.15.12:0/544528484 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef00ceb70 con 0x7f8ef00d1a10 2024-04-15T15:21:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.373+0000 7fc16659c640 1 -- 172.21.15.12:0/421725510 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc160053ce0 con 0x7fc1600ead00 2024-04-15T15:21:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f52e938e640 1 -- 172.21.15.12:0/890108721 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc02ce60 con 0x7f52e406fe00 2024-04-15T15:21:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f6dae200640 1 --2- 172.21.15.12:0/3043866186 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da815bf10 0x7f6da80b4af0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009e50 tx=0x7f6da002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=516078bd63ad3959 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f81effff640 1 --2- 172.21.15.12:0/3900808626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80df220 0x7f81e8038080 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009870 tx=0x7f81e402dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22422d75c4b5b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f3eef7fe640 1 --2- 172.21.15.12:0/2504694541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ee8006850 0x7f3ee8006cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8052890 tx=0x7f3ee002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d4061e8a6b74a7b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f6dac9fd640 1 -- 172.21.15.12:0/3043866186 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da002cb90 con 0x7f6da815bf10 2024-04-15T15:21:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f6dac9fd640 1 -- 172.21.15.12:0/3043866186 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da002cd30 con 0x7f6da815bf10 2024-04-15T15:21:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f6dac9fd640 1 -- 172.21.15.12:0/3043866186 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da0034610 con 0x7f6da815bf10 2024-04-15T15:21:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7f81eeffd640 1 -- 172.21.15.12:0/3900808626 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e403b070 con 0x7f81e80df220 2024-04-15T15:21:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f81eeffd640 1 -- 172.21.15.12:0/3900808626 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e402eae0 con 0x7f81e80df220 2024-04-15T15:21:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f81eeffd640 1 -- 172.21.15.12:0/3900808626 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4036440 con 0x7f81e80df220 2024-04-15T15:21:36.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7fc16659c640 1 --2- 172.21.15.12:0/421725510 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 0x7fc16004fa90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc1580098a0 tx=0x7fc15802b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c7d68e848f0601e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7fc5f159a640 1 -- 172.21.15.12:0/424519197 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4036070 con 0x7fc5f415a920 2024-04-15T15:21:36.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f3eedffb640 1 -- 172.21.15.12:0/2504694541 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee00315a0 con 0x7f3ee8006850 2024-04-15T15:21:36.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f81f52c0640 1 -- 172.21.15.12:0/3900808626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80df220 msgr2=0x7f81e8038080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f81f52c0640 1 --2- 172.21.15.12:0/3900808626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80df220 0x7f81e8038080 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009870 tx=0x7f81e402dd30 comp rx=0 tx=0).stop 2024-04-15T15:21:36.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f1ba9d9b640 1 --2- 172.21.15.12:0/1116176006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac0cf210 0x7f1bac0d2ba0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c009870 tx=0x7f1b9c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3001ff03e61377b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f1ba959a640 1 -- 172.21.15.12:0/1116176006 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c034070 con 0x7f1bac0cf210 2024-04-15T15:21:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f1ba959a640 1 -- 172.21.15.12:0/1116176006 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b9c030840 con 0x7f1bac0cf210 2024-04-15T15:21:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f1ba959a640 1 -- 172.21.15.12:0/1116176006 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c030c10 con 0x7f1bac0cf210 2024-04-15T15:21:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f3eedffb640 1 -- 172.21.15.12:0/2504694541 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee0037070 con 0x7f3ee8006850 2024-04-15T15:21:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f52ec5f4640 1 -- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52e406fe00 msgr2=0x7f52e40b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f52ec5f4640 1 --2- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52e406fe00 0x7f52e40b3470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009d90 tx=0x7f52dc02bec0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f52ec5f4640 1 -- 172.21.15.12:0/890108721 shutdown_connections 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f52ec5f4640 1 --2- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 0x7f52e40d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f52ec5f4640 1 --2- 172.21.15.12:0/890108721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52e406fe00 0x7f52e40b3470 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f52ec5f4640 1 -- 172.21.15.12:0/890108721 >> 172.21.15.12:0/890108721 conn(0x7f52e406be00 msgr2=0x7f52e406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f52ec5f4640 1 -- 172.21.15.12:0/890108721 shutdown_connections 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f52ec5f4640 1 -- 172.21.15.12:0/890108721 wait complete. 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f8eef7fe640 1 --2- 172.21.15.12:0/544528484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ef00d1a10 0x7f8ef006b9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00097a0 tx=0x7f8ee002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41c453be799bfaa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f8eedffb640 1 -- 172.21.15.12:0/544528484 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0037070 con 0x7f8ef00d1a10 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f8eedffb640 1 -- 172.21.15.12:0/544528484 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee0030710 con 0x7f8ef00d1a10 2024-04-15T15:21:36.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f8eedffb640 1 -- 172.21.15.12:0/544528484 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0030ae0 con 0x7f8ef00d1a10 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f6dafc63640 1 -- 172.21.15.12:0/3043866186 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da815bf10 msgr2=0x7f6da80b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f6dafc63640 1 --2- 172.21.15.12:0/3043866186 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da815bf10 0x7f6da80b4af0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009e50 tx=0x7f6da002bff0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7fc5f159a640 1 -- 172.21.15.12:0/424519197 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e4030d30 con 0x7fc5f415a920 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.374+0000 7fc5f159a640 1 -- 172.21.15.12:0/424519197 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e403a620 con 0x7fc5f415a920 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7fc5f8869640 1 -- 172.21.15.12:0/424519197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5f415a920 msgr2=0x7fc5f40b3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7fc5f8869640 1 --2- 172.21.15.12:0/424519197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5f415a920 0x7fc5f40b3160 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40b0e30 tx=0x7fc5e402ff80 comp rx=0 tx=0).stop 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7fc5f8869640 1 -- 172.21.15.12:0/424519197 shutdown_connections 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7fc5f8869640 1 --2- 172.21.15.12:0/424519197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5f415a920 0x7fc5f40b3160 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7fc5f8869640 1 -- 172.21.15.12:0/424519197 >> 172.21.15.12:0/424519197 conn(0x7fc5f40b5680 msgr2=0x7fc5f40b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7fc5f8869640 1 -- 172.21.15.12:0/424519197 shutdown_connections 2024-04-15T15:21:36.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7fc5f8869640 1 -- 172.21.15.12:0/424519197 wait complete. 2024-04-15T15:21:36.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f81f52c0640 1 -- 172.21.15.12:0/3900808626 shutdown_connections 2024-04-15T15:21:36.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f81f52c0640 1 --2- 172.21.15.12:0/3900808626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80df220 0x7f81e8038080 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f81f52c0640 1 -- 172.21.15.12:0/3900808626 >> 172.21.15.12:0/3900808626 conn(0x7f81e8009fb0 msgr2=0x7f81e800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f81f52c0640 1 -- 172.21.15.12:0/3900808626 shutdown_connections 2024-04-15T15:21:36.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f81f52c0640 1 -- 172.21.15.12:0/3900808626 wait complete. 2024-04-15T15:21:36.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.375+0000 7f3eedffb640 1 -- 172.21.15.12:0/2504694541 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee00315a0 con 0x7f3ee8006850 2024-04-15T15:21:36.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f3eeffff640 1 -- 172.21.15.12:0/2504694541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ee8006850 msgr2=0x7f3ee8006cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f3eeffff640 1 --2- 172.21.15.12:0/2504694541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ee8006850 0x7f3ee8006cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8052890 tx=0x7f3ee002b840 comp rx=0 tx=0).stop 2024-04-15T15:21:36.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f6dafc63640 1 -- 172.21.15.12:0/3043866186 shutdown_connections 2024-04-15T15:21:36.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f6dafc63640 1 --2- 172.21.15.12:0/3043866186 >> v2:172.21.15.12:3300/0 conn(0x7f6da80b3470 0x7f6da80d0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f6dafc63640 1 --2- 172.21.15.12:0/3043866186 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da815bf10 0x7f6da80b4af0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f6dafc63640 1 -- 172.21.15.12:0/3043866186 >> 172.21.15.12:0/3043866186 conn(0x7f6da806be00 msgr2=0x7f6da806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f8ef52b2640 1 -- 172.21.15.12:0/544528484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ef00d1a10 msgr2=0x7f8ef006b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f8ef52b2640 1 --2- 172.21.15.12:0/544528484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ef00d1a10 0x7f8ef006b9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00097a0 tx=0x7f8ee002b710 comp rx=0 tx=0).stop 2024-04-15T15:21:36.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7fc16559a640 1 -- 172.21.15.12:0/421725510 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158034070 con 0x7fc1600ead00 2024-04-15T15:21:36.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7fc16559a640 1 -- 172.21.15.12:0/421725510 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc158030860 con 0x7fc1600ead00 2024-04-15T15:21:36.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7fc16559a640 1 -- 172.21.15.12:0/421725510 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158030c70 con 0x7fc1600ead00 2024-04-15T15:21:36.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f3eeffff640 1 -- 172.21.15.12:0/2504694541 shutdown_connections 2024-04-15T15:21:36.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f3eeffff640 1 --2- 172.21.15.12:0/2504694541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ee8006850 0x7f3ee8006cc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f3eeffff640 1 --2- 172.21.15.12:0/2504694541 >> v2:172.21.15.12:3300/0 conn(0x7f3ee8005ed0 0x7f3ee8006310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f3eeffff640 1 -- 172.21.15.12:0/2504694541 >> 172.21.15.12:0/2504694541 conn(0x7f3ee8009fa0 msgr2=0x7f3ee800a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f8ef52b2640 1 -- 172.21.15.12:0/544528484 shutdown_connections 2024-04-15T15:21:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f8ef52b2640 1 --2- 172.21.15.12:0/544528484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ef00d1a10 0x7f8ef006b9e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f8ef52b2640 1 --2- 172.21.15.12:0/544528484 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 0x7f8ef00b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f8ef52b2640 1 -- 172.21.15.12:0/544528484 >> 172.21.15.12:0/544528484 conn(0x7f8ef009a9d0 msgr2=0x7f8ef00b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/1116176006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac0cf210 msgr2=0x7f1bac0d2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f1bb0ae4640 1 --2- 172.21.15.12:0/1116176006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac0cf210 0x7f1bac0d2ba0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c009870 tx=0x7f1b9c02b980 comp rx=0 tx=0).stop 2024-04-15T15:21:36.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.376+0000 7f8ef52b2640 1 -- 172.21.15.12:0/544528484 shutdown_connections 2024-04-15T15:21:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f3eeffff640 1 -- 172.21.15.12:0/2504694541 shutdown_connections 2024-04-15T15:21:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/1116176006 shutdown_connections 2024-04-15T15:21:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f1bb0ae4640 1 --2- 172.21.15.12:0/1116176006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 0x7f1bac0d30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f1bb0ae4640 1 --2- 172.21.15.12:0/1116176006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac0cf210 0x7f1bac0d2ba0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/1116176006 >> 172.21.15.12:0/1116176006 conn(0x7f1bac09d950 msgr2=0x7f1bac08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/1116176006 shutdown_connections 2024-04-15T15:21:36.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f52ec5f4640 1 Processor -- start 2024-04-15T15:21:36.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f6dafc63640 1 -- 172.21.15.12:0/3043866186 shutdown_connections 2024-04-15T15:21:36.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f52ec5f4640 1 -- start start 2024-04-15T15:21:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f6dafc63640 1 -- 172.21.15.12:0/3043866186 wait complete. 2024-04-15T15:21:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7fc16759e640 1 -- 172.21.15.12:0/421725510 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 msgr2=0x7fc16004fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7fc16759e640 1 --2- 172.21.15.12:0/421725510 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 0x7fc16004fa90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc1580098a0 tx=0x7fc15802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7fc5f8869640 1 Processor -- start 2024-04-15T15:21:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f8ef52b2640 1 -- 172.21.15.12:0/544528484 wait complete. 2024-04-15T15:21:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7fc5f8869640 1 -- start start 2024-04-15T15:21:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7f81f52c0640 1 Processor -- start 2024-04-15T15:21:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/1116176006 wait complete. 2024-04-15T15:21:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7f52ec5f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52e406fe00 0x7f52e416bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7fc16759e640 1 -- 172.21.15.12:0/421725510 shutdown_connections 2024-04-15T15:21:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7fc16759e640 1 --2- 172.21.15.12:0/421725510 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 0x7fc16004fa90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7fc16759e640 1 -- 172.21.15.12:0/421725510 >> 172.21.15.12:0/421725510 conn(0x7fc160009fb0 msgr2=0x7fc16000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7f81f52c0640 1 -- start start 2024-04-15T15:21:36.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.378+0000 7f8ef52b2640 1 Processor -- start 2024-04-15T15:21:36.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.377+0000 7f3eeffff640 1 -- 172.21.15.12:0/2504694541 wait complete. 2024-04-15T15:21:36.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7fc16759e640 1 -- 172.21.15.12:0/421725510 shutdown_connections 2024-04-15T15:21:36.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f3eeffff640 1 Processor -- start 2024-04-15T15:21:36.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7fc16759e640 1 -- 172.21.15.12:0/421725510 wait complete. 2024-04-15T15:21:36.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f1bb0ae4640 1 Processor -- start 2024-04-15T15:21:36.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f3eeffff640 1 -- start start 2024-04-15T15:21:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f1bb0ae4640 1 -- start start 2024-04-15T15:21:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f6dafc63640 1 Processor -- start 2024-04-15T15:21:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f52ec5f4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52e40af7f0 0x7f52e40ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7fc5f8869640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5f40b29c0 0x7fc5f414fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f52ec5f4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 0x7f52e40b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7fc5f8869640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5f40b3c00 0x7fc5f415e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f52ec5f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f52e40d0d00 con 0x7f52e406fe00 2024-04-15T15:21:36.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7fc5f8869640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5f415a920 0x7fc5f415e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f52ec5f4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f52e415c630 con 0x7f52e40af7f0 2024-04-15T15:21:36.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f52ec5f4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f52e40d0e80 con 0x7f52e415bf10 2024-04-15T15:21:36.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f52ea390640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52e406fe00 0x7f52e416bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f52ea390640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52e406fe00 0x7f52e416bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46698/0 (socket says 172.21.15.12:46698) 2024-04-15T15:21:36.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f52ea390640 1 -- 172.21.15.12:0/3549786125 learned_addr learned my addr 172.21.15.12:0/3549786125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f6dafc63640 1 -- start start 2024-04-15T15:21:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7fc5f259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5f40b29c0 0x7fc5f414fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.379+0000 7f8ef52b2640 1 -- start start 2024-04-15T15:21:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7fc5f259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5f40b29c0 0x7fc5f414fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:37572/0 (socket says 172.21.15.12:37572) 2024-04-15T15:21:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7fc5f2d9d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5f415a920 0x7fc5f415e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f52ea390640 1 -- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 msgr2=0x7f52e40b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f52ea390640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f52e415bf10 0x7f52e40b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f52ea390640 1 -- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52e40af7f0 msgr2=0x7f52e40ae6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:21:36.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f52ea390640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52e40af7f0 0x7f52e40ae6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f52ea390640 1 -- 172.21.15.12:0/3549786125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e40ce100 con 0x7f52e406fe00 2024-04-15T15:21:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7fc5f2d9d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5f415a920 0x7fc5f415e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33764/0 (socket says 172.21.15.12:33764) 2024-04-15T15:21:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f3eeffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ee8005ed0 0x7f3ee8032dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f1bb0ae4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac0cf210 0x7f1bac14d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f3eeffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8006850 0x7f3ee81884d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f1bb0ae4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 0x7f1bac156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baa59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac0cf210 0x7f1bac14d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baa59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac0cf210 0x7f1bac14d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46718/0 (socket says 172.21.15.12:46718) 2024-04-15T15:21:36.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1bb0ae4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac15e9a0 0x7f1bac157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f3eeffff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ee80eacf0 0x7f3ee8186880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f3eeffff640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee80547b0 con 0x7f3ee8006850 2024-04-15T15:21:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f3eeffff640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee8050190 con 0x7f3ee8005ed0 2024-04-15T15:21:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f3eee7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8006850 0x7f3ee81884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f3eee7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8006850 0x7f3ee81884d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46722/0 (socket says 172.21.15.12:46722) 2024-04-15T15:21:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f3eee7fc640 1 -- 172.21.15.12:0/2927468196 learned_addr learned my addr 172.21.15.12:0/2927468196 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee8054930 con 0x7f3ee80eacf0 2024-04-15T15:21:36.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f52ea390640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52e406fe00 0x7f52e416bd90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009eb0 tx=0x7f52dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dafc63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da80b3470 0x7f6da814c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dafc63640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6da80d07c0 0x7f6da81efb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dafc63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6da815bf10 0x7f6da80b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dafc63640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6da80afa60 con 0x7f6da815bf10 2024-04-15T15:21:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dafc63640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6da80b42f0 con 0x7f6da80d07c0 2024-04-15T15:21:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dafc63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6da80afbe0 con 0x7f6da80b3470 2024-04-15T15:21:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 0x7f1bac156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 0x7f1bac156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33776/0 (socket says 172.21.15.12:33776) 2024-04-15T15:21:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1bb0ae4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac0d2360 con 0x7f1bac0cf210 2024-04-15T15:21:36.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1bb0ae4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac06d2d0 con 0x7f1bac15e9a0 2024-04-15T15:21:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1bb0ae4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1bac0d24e0 con 0x7f1bac0cf690 2024-04-15T15:21:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baa59c640 1 -- 172.21.15.12:0/2398383495 learned_addr learned my addr 172.21.15.12:0/2398383495 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baad9d640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac15e9a0 0x7f1bac157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dad9ff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da80b3470 0x7f6da814c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dae200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6da815bf10 0x7f6da80b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dae200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6da815bf10 0x7f6da80b02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46728/0 (socket says 172.21.15.12:46728) 2024-04-15T15:21:36.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dae200640 1 -- 172.21.15.12:0/529041077 learned_addr learned my addr 172.21.15.12:0/529041077 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81f52c0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80501a0 0x7f81e804aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81f52c0640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f81e8053c70 0x7f81e80d9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81f52c0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81e80df220 0x7f81e80d9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81f52c0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8037a90 con 0x7f81e80501a0 2024-04-15T15:21:36.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81f52c0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8037910 con 0x7f81e80df220 2024-04-15T15:21:36.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dad1fe640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6da80d07c0 0x7f6da81efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7fc16759e640 1 Processor -- start 2024-04-15T15:21:36.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81f52c0640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f81e80df600 con 0x7f81e8053c70 2024-04-15T15:21:36.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f81e8053c70 0x7f81e80d9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7fc16759e640 1 -- start start 2024-04-15T15:21:36.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f81e8053c70 0x7f81e80d9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33802/0 (socket says 172.21.15.12:33802) 2024-04-15T15:21:36.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81ef7fe640 1 -- 172.21.15.12:0/3193608836 learned_addr learned my addr 172.21.15.12:0/3193608836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8ef52b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef00d08a0 0x7f8ef015e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8ef52b2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ef00d1a10 0x7f8ef0143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8ef52b2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 0x7f8ef0143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8ef52b2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef006bf20 con 0x7f8ef00d08a0 2024-04-15T15:21:36.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8ef52b2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef00d1010 con 0x7f8ef00d1a10 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8ef52b2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef006c0a0 con 0x7f8ef015b690 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8eee7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ef00d1a10 0x7f8ef0143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8eeeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef00d08a0 0x7f8ef015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8eeeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef00d08a0 0x7f8ef015e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46714/0 (socket says 172.21.15.12:46714) 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8eeeffd640 1 -- 172.21.15.12:0/2283979992 learned_addr learned my addr 172.21.15.12:0/2283979992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.380+0000 7f8eef7fe640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 0x7f8ef0143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f8eeeffd640 1 -- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 msgr2=0x7f8ef0143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f8eeeffd640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8ef015b690 0x7f8ef0143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f8eeeffd640 1 -- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ef00d1a10 msgr2=0x7f8ef0143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f8eeeffd640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ef00d1a10 0x7f8ef0143430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f8eeeffd640 1 -- 172.21.15.12:0/2283979992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef0142790 con 0x7f8ef00d08a0 2024-04-15T15:21:36.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baad9d640 1 -- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 msgr2=0x7f1bac156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baad9d640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1bac0cf690 0x7f1bac156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baad9d640 1 -- 172.21.15.12:0/2398383495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac0cf210 msgr2=0x7f1bac14d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baad9d640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bac0cf210 0x7f1bac14d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f1baad9d640 1 -- 172.21.15.12:0/2398383495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bac1e8c00 con 0x7f1bac15e9a0 2024-04-15T15:21:36.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f3eef7fe640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ee80eacf0 0x7f3ee8186880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dad9ff640 1 -- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6da80d07c0 msgr2=0x7f6da81efb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f6dad9ff640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6da80d07c0 0x7f6da81efb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f6dad9ff640 1 -- 172.21.15.12:0/529041077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6da815bf10 msgr2=0x7f6da80b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f6dad9ff640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6da815bf10 0x7f6da80b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f6dad9ff640 1 -- 172.21.15.12:0/529041077 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da814c8a0 con 0x7f6da80b3470 2024-04-15T15:21:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f8869640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f40b1b60 con 0x7fc5f40b3c00 2024-04-15T15:21:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f8869640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f40b19e0 con 0x7fc5f40b29c0 2024-04-15T15:21:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f8869640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f40b1ce0 con 0x7fc5f415a920 2024-04-15T15:21:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f2d9d640 1 -- 172.21.15.12:0/152704485 learned_addr learned my addr 172.21.15.12:0/152704485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f1baad9d640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac15e9a0 0x7f1bac157480 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400efc0 tx=0x7f1ba400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc044020 con 0x7f52e406fe00 2024-04-15T15:21:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc03f470 con 0x7f52e406fe00 2024-04-15T15:21:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc039040 con 0x7f52e406fe00 2024-04-15T15:21:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f1d9b640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5f40b3c00 0x7fc5f415e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f52ec5f4640 1 -- 172.21.15.12:0/3549786125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e4144e40 con 0x7f52e406fe00 2024-04-15T15:21:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f8eeeffd640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef00d08a0 0x7f8ef015e870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400d5f0 tx=0x7f8ee400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f6dad9ff640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da80b3470 0x7f6da814c070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d98000f60 tx=0x7f6d9800c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.381+0000 7f81effff640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80501a0 0x7f81e804aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4018020 con 0x7f1bac15e9a0 2024-04-15T15:21:36.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f3eef7fe640 1 -- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ee8005ed0 msgr2=0x7f3ee8032dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f3eef7fe640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ee8005ed0 0x7f3ee8032dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f3eef7fe640 1 -- 172.21.15.12:0/2927468196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8006850 msgr2=0x7f3ee81884d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f3eef7fe640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ee8006850 0x7f3ee81884d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f3eef7fe640 1 -- 172.21.15.12:0/2927468196 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee8033300 con 0x7f3ee80eacf0 2024-04-15T15:21:36.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f52ec5f4640 1 -- 172.21.15.12:0/3549786125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e40cc6a0 con 0x7f52e406fe00 2024-04-15T15:21:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f81ef7fe640 1 -- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81e80df220 msgr2=0x7f81e80d9c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:21:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f81ef7fe640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81e80df220 0x7f81e80d9c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f81ef7fe640 1 -- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80501a0 msgr2=0x7f81e804aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f81ef7fe640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80501a0 0x7f81e804aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f2d9d640 1 -- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5f40b29c0 msgr2=0x7fc5f414fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f2d9d640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5f40b29c0 0x7fc5f414fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f81ef7fe640 1 -- 172.21.15.12:0/3193608836 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e8178ca0 con 0x7f81e8053c70 2024-04-15T15:21:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f2d9d640 1 -- 172.21.15.12:0/152704485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5f40b3c00 msgr2=0x7fc5f415e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f2d9d640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5f40b3c00 0x7fc5f415e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc5f2d9d640 1 -- 172.21.15.12:0/152704485 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f41f9de0 con 0x7fc5f415a920 2024-04-15T15:21:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7fc16759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc160006ef0 0x7fc1600dd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f81effff640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e80501a0 0x7f81e804aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d980074e0 con 0x7f6da80b3470 2024-04-15T15:21:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d980047e0 con 0x7f6da80b3470 2024-04-15T15:21:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f3eef7fe640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ee80eacf0 0x7f3ee8186880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0033000 tx=0x7f3ee00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d98004c50 con 0x7f6da80b3470 2024-04-15T15:21:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc5f2d9d640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5f415a920 0x7fc5f415e580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00cb20 tx=0x7fc5ec002100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc16759e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc16004ffd0 0x7fc16004d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee00315a0 con 0x7f3ee80eacf0 2024-04-15T15:21:36.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc16659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc160006ef0 0x7fc1600dd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc16659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc160006ef0 0x7fc1600dd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46756/0 (socket says 172.21.15.12:46756) 2024-04-15T15:21:36.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec018070 con 0x7fc5f415a920 2024-04-15T15:21:36.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ec007e50 con 0x7fc5f415a920 2024-04-15T15:21:36.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec00ad80 con 0x7fc5f415a920 2024-04-15T15:21:36.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da814cb80 con 0x7f6da80b3470 2024-04-15T15:21:36.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc5f8869640 1 -- 172.21.15.12:0/152704485 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f41fa0c0 con 0x7fc5f415a920 2024-04-15T15:21:36.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee8187430 con 0x7f3ee80eacf0 2024-04-15T15:21:36.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee00326c0 con 0x7f3ee80eacf0 2024-04-15T15:21:36.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da8155230 con 0x7f6da80b3470 2024-04-15T15:21:36.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7fc5f8869640 1 -- 172.21.15.12:0/152704485 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f4150420 con 0x7fc5f415a920 2024-04-15T15:21:36.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bac1e8ee0 con 0x7f1bac15e9a0 2024-04-15T15:21:36.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.382+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bac1e94a0 con 0x7f1bac15e9a0 2024-04-15T15:21:36.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1bac0690f0 con 0x7f1bac15e9a0 2024-04-15T15:21:36.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba4014480 con 0x7f1bac15e9a0 2024-04-15T15:21:36.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f52ec5f4640 1 -- 172.21.15.12:0/3549786125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f52e4145360 con 0x7f52e406fe00 2024-04-15T15:21:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0032ab0 con 0x7f3ee80eacf0 2024-04-15T15:21:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee81879f0 con 0x7f3ee80eacf0 2024-04-15T15:21:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.383+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40125f0 con 0x7f8ef00d08a0 2024-04-15T15:21:36.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee4019070 con 0x7f8ef00d08a0 2024-04-15T15:21:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7fc16759e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 0x7fc16004d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7fc16759e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc160054940 con 0x7fc160006ef0 2024-04-15T15:21:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7fc16759e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc1600547c0 con 0x7fc16004ffd0 2024-04-15T15:21:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7fc16759e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc16004f550 con 0x7fc1600ead00 2024-04-15T15:21:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7fc16659c640 1 -- 172.21.15.12:0/2230259533 learned_addr learned my addr 172.21.15.12:0/2230259533 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3ee80052b0 con 0x7f3ee80eacf0 2024-04-15T15:21:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7fc5f8869640 1 -- 172.21.15.12:0/152704485 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc5d8001030 con 0x7fc5f415a920 2024-04-15T15:21:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7fc16659c640 1 -- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 msgr2=0x7fc16004d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7fc16659c640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1600ead00 0x7fc16004d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7fc16659c640 1 -- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc16004ffd0 msgr2=0x7fc16004d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7fc16659c640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc16004ffd0 0x7fc16004d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7fc16659c640 1 -- 172.21.15.12:0/2230259533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc160044720 con 0x7fc160006ef0 2024-04-15T15:21:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40129d0 con 0x7f8ef00d08a0 2024-04-15T15:21:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f81ef7fe640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f81e8053c70 0x7f81e80d9730 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00c410 tx=0x7f81dc00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc017070 con 0x7f81e8053c70 2024-04-15T15:21:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef015bd60 con 0x7f8ef00d08a0 2024-04-15T15:21:36.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef015c320 con 0x7f8ef00d08a0 2024-04-15T15:21:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7fc16659c640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc160006ef0 0x7fc1600dd4c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1580097f0 tx=0x7fc158004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6da80689e0 con 0x7f6da80b3470 2024-04-15T15:21:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc0035f0 con 0x7f81e8053c70 2024-04-15T15:21:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f81f52c0640 1 -- 172.21.15.12:0/3193608836 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e8174c00 con 0x7f81e8053c70 2024-04-15T15:21:36.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc017070 con 0x7f81e8053c70 2024-04-15T15:21:36.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.386+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4014870 con 0x7f1bac15e9a0 2024-04-15T15:21:36.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.386+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5ec01b070 con 0x7fc5f415a920 2024-04-15T15:21:36.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.384+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f52dc047020 con 0x7f52e406fe00 2024-04-15T15:21:36.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f52daffd640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f52c806c9f0 0x7f52c806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f52dc0ae610 con 0x7f52e406fe00 2024-04-15T15:21:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.386+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8ef00690f0 con 0x7f8ef00d08a0 2024-04-15T15:21:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f52e9b8f640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f52c806c9f0 0x7f52c806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f52daffd640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7f52c8076160 0x7f52c80785a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 --> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52dc009a50 con 0x7f52c8076160 2024-04-15T15:21:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.385+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f52dc0b5050 con 0x7f52e406fe00 2024-04-15T15:21:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.386+0000 7f52e9b8f640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f52c806c9f0 0x7f52c806eeb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f52e4099cf0 tx=0x7f52d40078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.386+0000 7f52eab91640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7f52c8076160 0x7f52c80785a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.386+0000 7f81f52c0640 1 -- 172.21.15.12:0/3193608836 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e81751c0 con 0x7f81e8053c70 2024-04-15T15:21:36.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.386+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158048070 con 0x7fc160006ef0 2024-04-15T15:21:36.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc5e2ffd640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc5d406c9c0 0x7fc5d406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 <== mon.1 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1ba4005d10 con 0x7f1bac15e9a0 2024-04-15T15:21:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc5ec08bda0 con 0x7fc5f415a920 2024-04-15T15:21:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc5e2ffd640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7fc5d4076130 0x7fc5d4078570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 --> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5ec021ec0 con 0x7fc5d4076130 2024-04-15T15:21:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc5ec08c190 con 0x7fc5f415a920 2024-04-15T15:21:36.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc5f1d9b640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7fc5d4076130 0x7fc5d4078570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81e81749f0 con 0x7f81e8053c70 2024-04-15T15:21:36.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ee4012b70 con 0x7f8ef00d08a0 2024-04-15T15:21:36.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f1b9affd640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b8c06c9c0 0x7f1b8c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7fc5f259c640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc5d406c9c0 0x7fc5d406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7fc5f1d9b640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7fc5d4076130 0x7fc5d4078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f1baa59c640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b8c06c9c0 0x7f1b8c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f8ed77fe640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8edc06c9f0 0x7f8edc06eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 <== mon.1 v2:172.21.15.55:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1ba408d4d0 con 0x7f1bac15e9a0 2024-04-15T15:21:36.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8ee408cb30 con 0x7f8ef00d08a0 2024-04-15T15:21:36.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7f52eab91640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7f52c8076160 0x7f52c80785a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f8ed77fe640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7f8edc076170 0x7f8edc0785b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 --> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ee4019070 con 0x7f8edc076170 2024-04-15T15:21:36.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8ee408cf20 con 0x7f8ef00d08a0 2024-04-15T15:21:36.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f1baa59c640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b8c06c9c0 0x7f1b8c06ee80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0007fd0 tx=0x7f1ba0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f8eee7fc640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8edc06c9f0 0x7f8edc06eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f1b9affd640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f1b8c076140 0x7f1b8c078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f81dc007020 con 0x7f81e8053c70 2024-04-15T15:21:36.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3ee004c060 con 0x7f3ee80eacf0 2024-04-15T15:21:36.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 --> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ba40020b0 con 0x7f1b8c076140 2024-04-15T15:21:36.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 <== mon.1 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1ba408d8c0 con 0x7f1bac15e9a0 2024-04-15T15:21:36.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 <== osd.4 v2:172.21.15.55:6808/3874250091 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f52dc009a50 con 0x7f52c8076160 2024-04-15T15:21:36.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.386+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1600449c0 con 0x7fc160006ef0 2024-04-15T15:21:36.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc158039d90 con 0x7fc160006ef0 2024-04-15T15:21:36.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158034040 con 0x7fc160006ef0 2024-04-15T15:21:36.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6d98011620 con 0x7f6da80b3470 2024-04-15T15:21:36.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f6da67fc640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6d9406c9c0 0x7f6d9406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f1ba9d9b640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f1b8c076140 0x7f1b8c078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.387+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1600e9b00 con 0x7fc160006ef0 2024-04-15T15:21:36.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc1600452b0 con 0x7fc160006ef0 2024-04-15T15:21:36.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.388+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6d9808f950 con 0x7f6da80b3470 2024-04-15T15:21:36.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f6da67fc640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f6d94076130 0x7f6d94078570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 --> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d94078c40 con 0x7f6d94076130 2024-04-15T15:21:36.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6d980942a0 con 0x7f6da80b3470 2024-04-15T15:21:36.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f6dad1fe640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6d9406c9c0 0x7f6d9406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f6dae200640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f6d94076130 0x7f6d94078570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f3ed77fe640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3edc06c9c0 0x7f3edc06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7fc5f259c640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc5d406c9c0 0x7fc5d406ee80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc5e403b000 tx=0x7fc5e4036000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f6dae200640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f6d94076130 0x7f6d94078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f81ecff9640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f81d406ccf0 0x7f81d406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 <== osd.0 v2:172.21.15.12:6800/3293425032 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc5ec021ec0 con 0x7fc5d4076130 2024-04-15T15:21:36.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f81dc0052e0 con 0x7f81e8053c70 2024-04-15T15:21:36.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc15804b070 con 0x7fc160006ef0 2024-04-15T15:21:36.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f1ba9d9b640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f1b8c076140 0x7f1b8c078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f6dad1fe640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6d9406c9c0 0x7f6d9406ee80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6da814d450 tx=0x7f6d9c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f81effff640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f81d406ccf0 0x7f81d406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f81ecff9640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7f81d40763f0 0x7f81d4078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 --> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81dc007650 con 0x7f81d40763f0 2024-04-15T15:21:36.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f81f4abf640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7f81d40763f0 0x7f81d4078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f81dc08b110 con 0x7f81e8053c70 2024-04-15T15:21:36.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3ee00ada10 con 0x7f3ee80eacf0 2024-04-15T15:21:36.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f3eeeffd640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3edc06c9c0 0x7f3edc06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 <== osd.6 v2:172.21.15.55:6816/2237244484 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d94078c40 con 0x7f6d94076130 2024-04-15T15:21:36.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.389+0000 7f8eee7fc640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8edc06c9f0 0x7f8edc06eeb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ef006bda0 tx=0x7f8ee8001a10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f8eef7fe640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7f8edc076170 0x7f8edc0785b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f8eef7fe640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7f8edc076170 0x7f8edc0785b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f3ed77fe640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7f3edc076140 0x7f3edc078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 --> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ee0043ec0 con 0x7f3edc076140 2024-04-15T15:21:36.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.390+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3ee00ade00 con 0x7f3ee80eacf0 2024-04-15T15:21:36.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.391+0000 7f81f4abf640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7f81d40763f0 0x7f81d4078830 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.391+0000 7f3eee7fc640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7f3edc076140 0x7f3edc078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.391+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 <== osd.2 v2:172.21.15.12:6808/2859338982 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ba40020b0 con 0x7f1b8c076140 2024-04-15T15:21:36.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.391+0000 7f3eeeffd640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3edc06c9c0 0x7f3edc06ee80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00687d0 tx=0x7f3ef006ddd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.391+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 <== osd.5 v2:172.21.15.55:6824/4105274290 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ee4019070 con 0x7f8edc076170 2024-04-15T15:21:36.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.391+0000 7fc156ffd640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc14806c8f0 0x7fc14806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.392+0000 7f81effff640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f81d406ccf0 0x7f81d406f1b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f81e4038000 tx=0x7f81e4032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.392+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 <== osd.1 v2:172.21.15.12:6816/1623484376 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81dc007650 con 0x7f81d40763f0 2024-04-15T15:21:36.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.391+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc1580adfc0 con 0x7fc160006ef0 2024-04-15T15:21:36.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.391+0000 7fc156ffd640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7fc148075ff0 0x7fc148078430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.392+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 --> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc158008f30 con 0x7fc148075ff0 2024-04-15T15:21:36.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.392+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc1580b05e0 con 0x7fc160006ef0 2024-04-15T15:21:36.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.392+0000 7fc165d9b640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc14806c8f0 0x7fc14806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.392+0000 7fc166d9d640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7fc148075ff0 0x7fc148078430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.394+0000 7fc165d9b640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc14806c8f0 0x7fc14806edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc16004eee0 tx=0x7fc15c0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.393+0000 7f3eee7fc640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7f3edc076140 0x7f3edc078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.394+0000 7fc166d9d640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7fc148075ff0 0x7fc148078430 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.394+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 <== osd.3 v2:172.21.15.12:6824/3520688318 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ee0043ec0 con 0x7f3edc076140 2024-04-15T15:21:36.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.395+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 <== osd.7 v2:172.21.15.55:6800/4101047654 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc158008f30 con 0x7fc148075ff0 2024-04-15T15:21:36.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.421+0000 7fc5f8869640 1 -- 172.21.15.12:0/152704485 --> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc5d8002d50 con 0x7fc5d4076130 2024-04-15T15:21:36.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.422+0000 7fc5e2ffd640 1 -- 172.21.15.12:0/152704485 <== osd.0 v2:172.21.15.12:6800/3293425032 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc5d8002d50 con 0x7fc5d4076130 2024-04-15T15:21:36.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.422+0000 7fc5e0ff9640 1 -- 172.21.15.12:0/152704485 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7fc5d4076130 msgr2=0x7fc5d4078570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.422+0000 7fc5e0ff9640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7fc5d4076130 0x7fc5d4078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7fc5e0ff9640 1 -- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc5d406c9c0 msgr2=0x7fc5d406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7fc5e0ff9640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc5d406c9c0 0x7fc5d406ee80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc5e403b000 tx=0x7fc5e4036000 comp rx=0 tx=0).stop 2024-04-15T15:21:36.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7f52ec5f4640 1 -- 172.21.15.12:0/3549786125 --> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f52e40689e0 con 0x7f52c8076160 2024-04-15T15:21:36.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7fc5e0ff9640 1 -- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5f415a920 msgr2=0x7fc5f415e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7fc5e0ff9640 1 --2- 172.21.15.12:0/152704485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5f415a920 0x7fc5f415e580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00cb20 tx=0x7fc5ec002100 comp rx=0 tx=0).stop 2024-04-15T15:21:36.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7fc5f2d9d640 1 -- 172.21.15.12:0/152704485 reap_dead start 2024-04-15T15:21:36.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7fc5e0ff9640 1 -- 172.21.15.12:0/152704485 shutdown_connections 2024-04-15T15:21:36.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7fc5e0ff9640 1 -- 172.21.15.12:0/152704485 >> 172.21.15.12:0/152704485 conn(0x7fc5f40b5680 msgr2=0x7fc5f40ad140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.423+0000 7f52daffd640 1 -- 172.21.15.12:0/3549786125 <== osd.4 v2:172.21.15.55:6808/3874250091 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f52e40689e0 con 0x7f52c8076160 2024-04-15T15:21:36.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 --> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6da80ad2f0 con 0x7f6d94076130 2024-04-15T15:21:36.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7fc5e0ff9640 1 -- 172.21.15.12:0/152704485 shutdown_connections 2024-04-15T15:21:36.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7fc5e0ff9640 1 -- 172.21.15.12:0/152704485 wait complete. 2024-04-15T15:21:36.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 --> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f81e80052c0 con 0x7f81d40763f0 2024-04-15T15:21:36.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f52d8ff9640 1 -- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7f52c8076160 msgr2=0x7f52c80785a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f52d8ff9640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7f52c8076160 0x7f52c80785a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 --> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1bac1e9ef0 con 0x7f1b8c076140 2024-04-15T15:21:36.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f6da67fc640 1 -- 172.21.15.12:0/529041077 <== osd.6 v2:172.21.15.55:6816/2237244484 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6da80ad2f0 con 0x7f6d94076130 2024-04-15T15:21:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f81ecff9640 1 -- 172.21.15.12:0/3193608836 <== osd.1 v2:172.21.15.12:6816/1623484376 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f81e80052c0 con 0x7f81d40763f0 2024-04-15T15:21:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f52d8ff9640 1 -- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f52c806c9f0 msgr2=0x7f52c806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f52d8ff9640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f52c806c9f0 0x7f52c806eeb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f52e4099cf0 tx=0x7f52d40078f0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f6d94076130 msgr2=0x7f6d94078570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f1b9affd640 1 -- 172.21.15.12:0/2398383495 <== osd.2 v2:172.21.15.12:6808/2859338982 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1b9c030810 con 0x7f1b8c076140 2024-04-15T15:21:36.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f52d8ff9640 1 -- 172.21.15.12:0/3549786125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52e406fe00 msgr2=0x7f52e416bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f6dafc63640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f6d94076130 0x7f6d94078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.424+0000 7f52d8ff9640 1 --2- 172.21.15.12:0/3549786125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52e406fe00 0x7f52e416bd90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009eb0 tx=0x7f52dc004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:36.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f52eab91640 1 -- 172.21.15.12:0/3549786125 reap_dead start 2024-04-15T15:21:36.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f52d8ff9640 1 -- 172.21.15.12:0/3549786125 shutdown_connections 2024-04-15T15:21:36.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f52d8ff9640 1 -- 172.21.15.12:0/3549786125 >> 172.21.15.12:0/3549786125 conn(0x7f52e406be00 msgr2=0x7f52e40b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 --> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8ef00af130 con 0x7f8edc076170 2024-04-15T15:21:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6d9406c9c0 msgr2=0x7f6d9406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dafc63640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6d9406c9c0 0x7f6d9406ee80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6da814d450 tx=0x7f6d9c008040 comp rx=0 tx=0).stop 2024-04-15T15:21:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da80b3470 msgr2=0x7f6da814c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dafc63640 1 --2- 172.21.15.12:0/529041077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6da80b3470 0x7f6da814c070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d98000f60 tx=0x7f6d9800c890 comp rx=0 tx=0).stop 2024-04-15T15:21:36.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dae200640 1 -- 172.21.15.12:0/529041077 reap_dead start 2024-04-15T15:21:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f1b8c076140 msgr2=0x7f1b8c078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 shutdown_connections 2024-04-15T15:21:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 >> 172.21.15.12:0/529041077 conn(0x7f6da806be00 msgr2=0x7f6da814fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f1bb0ae4640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f1b8c076140 0x7f1b8c078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f52d8ff9640 1 -- 172.21.15.12:0/3549786125 shutdown_connections 2024-04-15T15:21:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f52d8ff9640 1 -- 172.21.15.12:0/3549786125 wait complete. 2024-04-15T15:21:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b8c06c9c0 msgr2=0x7f1b8c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f1bb0ae4640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b8c06c9c0 0x7f1b8c06ee80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0007fd0 tx=0x7f1ba0009040 comp rx=0 tx=0).stop 2024-04-15T15:21:36.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 shutdown_connections 2024-04-15T15:21:36.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f8ed77fe640 1 -- 172.21.15.12:0/2283979992 <== osd.5 v2:172.21.15.55:6824/4105274290 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8ef00af130 con 0x7f8edc076170 2024-04-15T15:21:36.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f6dafc63640 1 -- 172.21.15.12:0/529041077 wait complete. 2024-04-15T15:21:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac15e9a0 msgr2=0x7f1bac157480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f1bb0ae4640 1 --2- 172.21.15.12:0/2398383495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bac15e9a0 0x7f1bac157480 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400efc0 tx=0x7f1ba400c490 comp rx=0 tx=0).stop 2024-04-15T15:21:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.425+0000 7f1baad9d640 1 -- 172.21.15.12:0/2398383495 reap_dead start 2024-04-15T15:21:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.426+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 shutdown_connections 2024-04-15T15:21:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.426+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 >> 172.21.15.12:0/2398383495 conn(0x7f1bac09d950 msgr2=0x7f1bac0b60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.426+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 shutdown_connections 2024-04-15T15:21:36.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.426+0000 7f1bb0ae4640 1 -- 172.21.15.12:0/2398383495 wait complete. 2024-04-15T15:21:36.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.426+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7f81d40763f0 msgr2=0x7f81d4078830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.426+0000 7f81da7fc640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7f81d40763f0 0x7f81d4078830 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f81d406ccf0 msgr2=0x7f81d406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f81da7fc640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f81d406ccf0 0x7f81d406f1b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f81e4038000 tx=0x7f81e4032000 comp rx=0 tx=0).stop 2024-04-15T15:21:36.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f81e8053c70 msgr2=0x7f81e80d9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f81da7fc640 1 --2- 172.21.15.12:0/3193608836 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f81e8053c70 0x7f81e80d9730 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00c410 tx=0x7f81dc00c8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f81f4abf640 1 -- 172.21.15.12:0/3193608836 reap_dead start 2024-04-15T15:21:36.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 --> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc160047270 con 0x7fc148075ff0 2024-04-15T15:21:36.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 shutdown_connections 2024-04-15T15:21:36.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 >> 172.21.15.12:0/3193608836 conn(0x7f81e8009fb0 msgr2=0x7f81e8036590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 --> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3ee8033530 con 0x7f3edc076140 2024-04-15T15:21:36.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7f8edc076170 msgr2=0x7f8edc0785b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8ef52b2640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7f8edc076170 0x7f8edc0785b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 shutdown_connections 2024-04-15T15:21:36.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8edc06c9f0 msgr2=0x7f8edc06eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8ef52b2640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8edc06c9f0 0x7f8edc06eeb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ef006bda0 tx=0x7f8ee8001a10 comp rx=0 tx=0).stop 2024-04-15T15:21:36.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef00d08a0 msgr2=0x7f8ef015e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8ef52b2640 1 --2- 172.21.15.12:0/2283979992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef00d08a0 0x7f8ef015e870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400d5f0 tx=0x7f8ee400dab0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8eef7fe640 1 -- 172.21.15.12:0/2283979992 reap_dead start 2024-04-15T15:21:36.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f3ed77fe640 1 -- 172.21.15.12:0/2927468196 <== osd.3 v2:172.21.15.12:6824/3520688318 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3ee8033530 con 0x7f3edc076140 2024-04-15T15:21:36.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 shutdown_connections 2024-04-15T15:21:36.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.427+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 >> 172.21.15.12:0/2283979992 conn(0x7f8ef009a9d0 msgr2=0x7f8ef00af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7f81da7fc640 1 -- 172.21.15.12:0/3193608836 wait complete. 2024-04-15T15:21:36.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 shutdown_connections 2024-04-15T15:21:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7fc156ffd640 1 -- 172.21.15.12:0/2230259533 <== osd.7 v2:172.21.15.55:6800/4101047654 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc160047270 con 0x7fc148075ff0 2024-04-15T15:21:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7f8ef52b2640 1 -- 172.21.15.12:0/2283979992 wait complete. 2024-04-15T15:21:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7f3edc076140 msgr2=0x7f3edc078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7f3eeffff640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7f3edc076140 0x7f3edc078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7fc148075ff0 msgr2=0x7fc148078430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7fc16759e640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7fc148075ff0 0x7fc148078430 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3edc06c9c0 msgr2=0x7f3edc06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eeffff640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3edc06c9c0 0x7f3edc06ee80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00687d0 tx=0x7f3ef006ddd0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ee80eacf0 msgr2=0x7f3ee8186880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.428+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc14806c8f0 msgr2=0x7fc14806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eeffff640 1 --2- 172.21.15.12:0/2927468196 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ee80eacf0 0x7f3ee8186880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0033000 tx=0x7f3ee00046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7fc16759e640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc14806c8f0 0x7fc14806edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc16004eee0 tx=0x7fc15c0079b0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eef7fe640 1 -- 172.21.15.12:0/2927468196 reap_dead start 2024-04-15T15:21:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 shutdown_connections 2024-04-15T15:21:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 >> 172.21.15.12:0/2927468196 conn(0x7f3ee8009fa0 msgr2=0x7f3ee804f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc160006ef0 msgr2=0x7fc1600dd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7fc16759e640 1 --2- 172.21.15.12:0/2230259533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc160006ef0 0x7fc1600dd4c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1580097f0 tx=0x7fc158004730 comp rx=0 tx=0).stop 2024-04-15T15:21:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7fc166d9d640 1 -- 172.21.15.12:0/2230259533 reap_dead start 2024-04-15T15:21:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 shutdown_connections 2024-04-15T15:21:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.429+0000 7f3eeffff640 1 -- 172.21.15.12:0/2927468196 wait complete. 2024-04-15T15:21:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.430+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 shutdown_connections 2024-04-15T15:21:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.430+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 >> 172.21.15.12:0/2230259533 conn(0x7fc160009fb0 msgr2=0x7fc160036430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.430+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 shutdown_connections 2024-04-15T15:21:36.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.430+0000 7fc16759e640 1 -- 172.21.15.12:0/2230259533 wait complete. 2024-04-15T15:21:36.498 INFO:teuthology.orchestra.run.smithi012.stdout:55834574858 2024-04-15T15:21:36.499 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-15T15:21:36.500 INFO:teuthology.orchestra.run.smithi012.stdout:55834574858 2024-04-15T15:21:36.500 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-15T15:21:36.501 INFO:teuthology.orchestra.run.smithi012.stdout:55834574859 2024-04-15T15:21:36.501 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-15T15:21:36.502 INFO:teuthology.orchestra.run.smithi012.stdout:55834574859 2024-04-15T15:21:36.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-15T15:21:36.503 INFO:teuthology.orchestra.run.smithi012.stdout:55834574858 2024-04-15T15:21:36.503 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-15T15:21:36.504 INFO:teuthology.orchestra.run.smithi012.stdout:55834574860 2024-04-15T15:21:36.504 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-15T15:21:36.505 INFO:teuthology.orchestra.run.smithi012.stdout:55834574859 2024-04-15T15:21:36.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-15T15:21:36.506 INFO:teuthology.orchestra.run.smithi012.stdout:55834574858 2024-04-15T15:21:36.506 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-15T15:21:36.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.716+0000 7f6c4afe2640 1 Processor -- start 2024-04-15T15:21:36.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.716+0000 7f6c4afe2640 1 -- start start 2024-04-15T15:21:36.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7faecbb67640 1 Processor -- start 2024-04-15T15:21:36.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7f6c4afe2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 0x7f6c4416e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7f6c4afe2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c4416d820 con 0x7f6c4416ef60 2024-04-15T15:21:36.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7faecbb67640 1 -- start start 2024-04-15T15:21:36.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7f2e4493c640 1 Processor -- start 2024-04-15T15:21:36.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7f6c4afe2640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c4416db00 con 0x7f6c440b36b0 2024-04-15T15:21:36.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7f6c4afe2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6c4416dde0 con 0x7f6c4416d1f0 2024-04-15T15:21:36.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7f6c48d7e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6c4416ef60 0x7f6c4416f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56326/0 (socket says 172.21.15.12:56326) 2024-04-15T15:21:36.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7f6c48d7e640 1 -- 172.21.15.12:0/709370207 learned_addr learned my addr 172.21.15.12:0/709370207 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:36.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7f6c43fff640 1 --2- 172.21.15.12:0/709370207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 0x7f6c4416e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.717+0000 7f2e4493c640 1 -- start start 2024-04-15T15:21:36.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7f6c437fe640 1 -- 172.21.15.12:0/709370207 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887688132 0 0) 0x7f6c4416db00 con 0x7f6c440b36b0 2024-04-15T15:21:36.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7f6c437fe640 1 -- 172.21.15.12:0/709370207 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c2c003520 con 0x7f6c440b36b0 2024-04-15T15:21:36.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7f6c43fff640 1 -- 172.21.15.12:0/709370207 >> v1:172.21.15.55:6789/0 conn(0x7f6c440b36b0 legacy=0x7f6c441695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7f6c43fff640 1 -- 172.21.15.12:0/709370207 >> v1:172.21.15.12:6789/0 conn(0x7f6c4416ef60 legacy=0x7f6c4416f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7f6c43fff640 1 -- 172.21.15.12:0/709370207 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c4416e0c0 con 0x7f6c4416d1f0 2024-04-15T15:21:36.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7fc8df59e640 1 Processor -- start 2024-04-15T15:21:36.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7faecbb67640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faec415dd30 0x7faec406f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7faecbb67640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec40cd540 0x7faec406da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7faecbb67640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faec40cf200 con 0x7faec40737e0 2024-04-15T15:21:36.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7faecbb67640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7faec40cf4e0 con 0x7faec415dd30 2024-04-15T15:21:36.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7faecbb67640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faec40cf7c0 con 0x7faec40cd540 2024-04-15T15:21:36.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7fc8df59e640 1 -- start start 2024-04-15T15:21:36.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7f2e4493c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2e38006a30 0x7f2e38006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7faec9903640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faec415dd30 0x7faec406f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7faec9903640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faec415dd30 0x7faec406f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46772/0 (socket says 172.21.15.12:46772) 2024-04-15T15:21:36.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7faec9903640 1 -- 172.21.15.12:0/128292775 learned_addr learned my addr 172.21.15.12:0/128292775 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7f2e3ed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2e380ead00 0x7f2e3804fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56356/0 (socket says 172.21.15.12:56356) 2024-04-15T15:21:36.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7faeca104640 1 --2- 172.21.15.12:0/128292775 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec40cd540 0x7faec406da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7f2e3ed9d640 1 -- 172.21.15.12:0/342837745 learned_addr learned my addr 172.21.15.12:0/342837745 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7f2e4493c640 1 -- 172.21.15.12:0/342837745 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e38053720 con 0x7f2e380ead00 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7f2e4493c640 1 -- 172.21.15.12:0/342837745 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e38053a00 con 0x7f2e38005ee0 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7f2e4493c640 1 -- 172.21.15.12:0/342837745 --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2e38054330 con 0x7f2e38006a30 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.719+0000 7f2e3f59e640 1 --2- 172.21.15.12:0/342837745 >> v2:172.21.15.55:3300/0 conn(0x7f2e38006a30 0x7f2e38006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7f2e3dd9b640 1 -- 172.21.15.12:0/342837745 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 10897731 0 0) 0x7f2e38053a00 con 0x7f2e38005ee0 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7f2e3dd9b640 1 -- 172.21.15.12:0/342837745 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e28003520 con 0x7f2e38005ee0 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7f2e3f59e640 1 -- 172.21.15.12:0/342837745 >> v1:172.21.15.55:6789/0 conn(0x7f2e38005ee0 legacy=0x7f2e38006320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7f2e3f59e640 1 -- 172.21.15.12:0/342837745 >> v1:172.21.15.12:6789/0 conn(0x7f2e380ead00 legacy=0x7f2e3804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7f2e3f59e640 1 -- 172.21.15.12:0/342837745 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e38053ce0 con 0x7f2e38006a30 2024-04-15T15:21:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8df59e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc8e00d0d20 0x7fc8e00d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8df59e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 0x7fc8e00d2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8df59e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e00cef60 con 0x7fc8e015e680 2024-04-15T15:21:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8df59e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc8e00d1770 con 0x7fc8e00d0d20 2024-04-15T15:21:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8df59e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e00d1a50 con 0x7fc8e00d11b0 2024-04-15T15:21:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8d7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc8e00d0d20 0x7fc8e00d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8d7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc8e00d0d20 0x7fc8e00d29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46786/0 (socket says 172.21.15.12:46786) 2024-04-15T15:21:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8d7fff640 1 -- 172.21.15.12:0/534816233 learned_addr learned my addr 172.21.15.12:0/534816233 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7faec9903640 1 -- 172.21.15.12:0/128292775 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec40cd540 msgr2=0x7faec406da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7faec9903640 1 --2- 172.21.15.12:0/128292775 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec40cd540 0x7faec406da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8d7fff640 1 -- 172.21.15.12:0/534816233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 msgr2=0x7fc8e00d2ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:21:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fc8d7fff640 1 --2- 172.21.15.12:0/534816233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 0x7fc8e00d2ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fc8d7fff640 1 -- 172.21.15.12:0/534816233 >> v1:172.21.15.12:6789/0 conn(0x7fc8e015e680 legacy=0x7fc8e006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fc8d7fff640 1 -- 172.21.15.12:0/534816233 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e00cf200 con 0x7fc8e00d0d20 2024-04-15T15:21:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7faec9903640 1 -- 172.21.15.12:0/128292775 >> v1:172.21.15.12:6789/0 conn(0x7faec40737e0 legacy=0x7faec40cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7faec9903640 1 -- 172.21.15.12:0/128292775 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec40cfaa0 con 0x7faec415dd30 2024-04-15T15:21:36.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7f2e3f59e640 1 --2- 172.21.15.12:0/342837745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e38006a30 0x7f2e38006ea0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2e300097a0 tx=0x7f2e3002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c39cb94e3e23d3af server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7f2e3dd9b640 1 -- 172.21.15.12:0/342837745 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30035020 con 0x7f2e38006a30 2024-04-15T15:21:36.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7faec9903640 1 --2- 172.21.15.12:0/128292775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec415dd30 0x7faec406f100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faebc00d370 tx=0x7faebc02e280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d217cfc26a35fdb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7f2e3dd9b640 1 -- 172.21.15.12:0/342837745 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e30031830 con 0x7f2e38006a30 2024-04-15T15:21:36.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7f2e3dd9b640 1 -- 172.21.15.12:0/342837745 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30031be0 con 0x7f2e38006a30 2024-04-15T15:21:36.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7f5fc8a88640 1 Processor -- start 2024-04-15T15:21:36.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fc8d7fff640 1 --2- 172.21.15.12:0/534816233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e00d0d20 0x7fc8e00d29a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80098a0 tx=0x7fc8c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=301d6fe7f8c6c5b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7f5fc8a88640 1 -- start start 2024-04-15T15:21:36.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fc8ddd9b640 1 -- 172.21.15.12:0/534816233 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8035020 con 0x7fc8e00d0d20 2024-04-15T15:21:36.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7fa3abeb1640 1 Processor -- start 2024-04-15T15:21:36.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.718+0000 7fa3abeb1640 1 -- start start 2024-04-15T15:21:36.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fa3abeb1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa3a406c9f0 0x7fa3a406cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fa3abeb1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a406d310 con 0x7fa3a41643e0 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fa3abeb1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a4163ac0 con 0x7fa3a406fd80 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fa3abeb1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a40cd790 con 0x7fa3a406c9f0 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fa3a944c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa3a406fd80 0x7fa3a4163220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:58858/0 (socket says 172.21.15.12:58858) 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fa3a944c640 1 -- 172.21.15.12:0/691695333 learned_addr learned my addr 172.21.15.12:0/691695333 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.720+0000 7fa3a9c4d640 1 --2- 172.21.15.12:0/691695333 >> v2:172.21.15.12:3300/0 conn(0x7fa3a406c9f0 0x7fa3a406cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2179962682 0 0) 0x7fa3a406d310 con 0x7fa3a41643e0 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa390003520 con 0x7fa3a41643e0 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 11257173 0 0) 0x7fa3a4163ac0 con 0x7fa3a406fd80 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3a406d310 con 0x7fa3a406fd80 2024-04-15T15:21:36.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7f6c43fff640 1 --2- 172.21.15.12:0/709370207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 0x7f6c4416e8c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6c340097d0 tx=0x7f6c3402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc5f17d51cb64eb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3248786710 0 0) 0x7fa390003520 con 0x7fa3a41643e0 2024-04-15T15:21:36.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 >> v2:172.21.15.12:3300/0 conn(0x7fa3a406c9f0 msgr2=0x7fa3a406cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 --2- 172.21.15.12:0/691695333 >> v2:172.21.15.12:3300/0 conn(0x7fa3a406c9f0 0x7fa3a406cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 >> v1:172.21.15.55:6789/0 conn(0x7fa3a406fd80 legacy=0x7fa3a4163220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a4163da0 con 0x7fa3a41643e0 2024-04-15T15:21:36.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fa39c0019f0 con 0x7fa3a41643e0 2024-04-15T15:21:36.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 >> v1:172.21.15.12:6789/0 conn(0x7fa3a41643e0 legacy=0x7fa3a40b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa390003b90 0x7fa390003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.721+0000 7fa3a8c4b640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa390004800 0x7fa390004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3a8c4b640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3900051c0 0x7fa39002d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3a944c640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa390004800 0x7fa390004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa39002dfc0 con 0x7fa390003b90 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3aa44e640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3900051c0 0x7fa39002d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa39002e170 con 0x7fa390004800 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa39002e370 con 0x7fa3900051c0 2024-04-15T15:21:36.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3a9c4d640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa390003b90 0x7fa390003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fc8ddd9b640 1 -- 172.21.15.12:0/534816233 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c8031870 con 0x7fc8e00d0d20 2024-04-15T15:21:36.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fc8ddd9b640 1 -- 172.21.15.12:0/534816233 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8031c40 con 0x7fc8e00d0d20 2024-04-15T15:21:36.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3aa44e640 1 -- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa390004800 msgr2=0x7fa390004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3aa44e640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa390004800 0x7fa390004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3aa44e640 1 -- 172.21.15.12:0/691695333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa390003b90 msgr2=0x7fa390003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3aa44e640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa390003b90 0x7fa390003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7fa3aa44e640 1 -- 172.21.15.12:0/691695333 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa390003520 con 0x7fa3900051c0 2024-04-15T15:21:36.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7f2e4493c640 1 -- 172.21.15.12:0/342837745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e38006a30 msgr2=0x7f2e38006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7faec8901640 1 -- 172.21.15.12:0/128292775 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc034070 con 0x7faec415dd30 2024-04-15T15:21:36.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.722+0000 7f2e4493c640 1 --2- 172.21.15.12:0/342837745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e38006a30 0x7f2e38006ea0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2e300097a0 tx=0x7f2e3002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7faec8901640 1 -- 172.21.15.12:0/128292775 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faebc038600 con 0x7faec415dd30 2024-04-15T15:21:36.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7f2e4493c640 1 -- 172.21.15.12:0/342837745 shutdown_connections 2024-04-15T15:21:36.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7f2e4493c640 1 --2- 172.21.15.12:0/342837745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e38006a30 0x7f2e38006ea0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7f2e4493c640 1 -- 172.21.15.12:0/342837745 >> 172.21.15.12:0/342837745 conn(0x7f2e38009fb0 msgr2=0x7f2e3800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7faec8901640 1 -- 172.21.15.12:0/128292775 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc00be60 con 0x7faec415dd30 2024-04-15T15:21:36.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7f520d603640 1 Processor -- start 2024-04-15T15:21:36.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7fc8df59e640 1 -- 172.21.15.12:0/534816233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e00d0d20 msgr2=0x7fc8e00d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7fc8df59e640 1 --2- 172.21.15.12:0/534816233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e00d0d20 0x7fc8e00d29a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80098a0 tx=0x7fc8c802b910 comp rx=0 tx=0).stop 2024-04-15T15:21:36.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7fa3aa44e640 1 -- 172.21.15.12:0/691695333 reap_dead start 2024-04-15T15:21:36.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7fc8df59e640 1 -- 172.21.15.12:0/534816233 shutdown_connections 2024-04-15T15:21:36.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7fc8df59e640 1 --2- 172.21.15.12:0/534816233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 0x7fc8e00d2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7fc8df59e640 1 --2- 172.21.15.12:0/534816233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e00d0d20 0x7fc8e00d29a0 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80098a0 tx=0x7fc8c802b910 comp rx=0 tx=0).stop 2024-04-15T15:21:36.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7fc8df59e640 1 -- 172.21.15.12:0/534816233 >> 172.21.15.12:0/534816233 conn(0x7fc8e009d740 msgr2=0x7fc8e008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7fa3a9c4d640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa390003b90 0x7fa390003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:36.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7faecbb67640 1 -- 172.21.15.12:0/128292775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec415dd30 msgr2=0x7faec406f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7f6c437fe640 1 -- 172.21.15.12:0/709370207 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34035020 con 0x7f6c4416d1f0 2024-04-15T15:21:36.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7f6c437fe640 1 -- 172.21.15.12:0/709370207 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c34031800 con 0x7f6c4416d1f0 2024-04-15T15:21:36.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7f6c437fe640 1 -- 172.21.15.12:0/709370207 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34031bd0 con 0x7f6c4416d1f0 2024-04-15T15:21:36.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7faecbb67640 1 --2- 172.21.15.12:0/128292775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec415dd30 0x7faec406f100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faebc00d370 tx=0x7faebc02e280 comp rx=0 tx=0).stop 2024-04-15T15:21:36.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc8a88640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc415a620 0x7f5fc415aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f2e4493c640 1 -- 172.21.15.12:0/342837745 shutdown_connections 2024-04-15T15:21:36.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7faecbb67640 1 -- 172.21.15.12:0/128292775 shutdown_connections 2024-04-15T15:21:36.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7faecbb67640 1 --2- 172.21.15.12:0/128292775 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec40cd540 0x7faec406da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7faecbb67640 1 --2- 172.21.15.12:0/128292775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec415dd30 0x7faec406f100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7faecbb67640 1 -- 172.21.15.12:0/128292775 >> 172.21.15.12:0/128292775 conn(0x7faec40b1c50 msgr2=0x7faec40b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc8a88640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5fc415b810 0x7f5fc40d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7fa3aa44e640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3900051c0 0x7fa39002d9d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c4c0 tx=0x7fa390005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae23c620a4bf78e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f2e4493c640 1 -- 172.21.15.12:0/342837745 wait complete. 2024-04-15T15:21:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7faecbb67640 1 -- 172.21.15.12:0/128292775 shutdown_connections 2024-04-15T15:21:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc259c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc415a620 0x7f5fc415aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc259c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc415a620 0x7f5fc415aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33840/0 (socket says 172.21.15.12:33840) 2024-04-15T15:21:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc259c640 1 -- 172.21.15.12:0/2278870485 learned_addr learned my addr 172.21.15.12:0/2278870485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc8a88640 1 -- 172.21.15.12:0/2278870485 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fc406ea10 con 0x7f5fc40cff20 2024-04-15T15:21:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc8a88640 1 -- 172.21.15.12:0/2278870485 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5fc415bc80 con 0x7f5fc415b810 2024-04-15T15:21:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7fc8df59e640 1 -- 172.21.15.12:0/534816233 shutdown_connections 2024-04-15T15:21:36.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.723+0000 7f520d603640 1 -- start start 2024-04-15T15:21:36.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc8a88640 1 -- 172.21.15.12:0/2278870485 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5fc406e7b0 con 0x7f5fc415a620 2024-04-15T15:21:36.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7f5fc2d9d640 1 --2- 172.21.15.12:0/2278870485 >> v2:172.21.15.12:3300/0 conn(0x7f5fc415b810 0x7f5fc40d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7faecbb67640 1 -- 172.21.15.12:0/128292775 wait complete. 2024-04-15T15:21:36.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f6c4afe2640 1 -- 172.21.15.12:0/709370207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 msgr2=0x7f6c4416e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f6c4afe2640 1 --2- 172.21.15.12:0/709370207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 0x7f6c4416e8c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6c340097d0 tx=0x7f6c3402b8a0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f5fc259c640 1 -- 172.21.15.12:0/2278870485 >> v2:172.21.15.12:3300/0 conn(0x7f5fc415b810 msgr2=0x7f5fc40d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f5fc259c640 1 --2- 172.21.15.12:0/2278870485 >> v2:172.21.15.12:3300/0 conn(0x7f5fc415b810 0x7f5fc40d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f5fc259c640 1 -- 172.21.15.12:0/2278870485 >> v1:172.21.15.55:6789/0 conn(0x7f5fc40cff20 legacy=0x7f5fc406e0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:21:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f5fc259c640 1 -- 172.21.15.12:0/2278870485 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc406ecf0 con 0x7f5fc415a620 2024-04-15T15:21:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f6c4afe2640 1 -- 172.21.15.12:0/709370207 shutdown_connections 2024-04-15T15:21:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f6c4afe2640 1 --2- 172.21.15.12:0/709370207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 0x7f6c4416e8c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f6c4afe2640 1 -- 172.21.15.12:0/709370207 >> 172.21.15.12:0/709370207 conn(0x7f6c440b5680 msgr2=0x7f6c440b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f6c4afe2640 1 -- 172.21.15.12:0/709370207 shutdown_connections 2024-04-15T15:21:36.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.724+0000 7fc8df59e640 1 -- 172.21.15.12:0/534816233 wait complete. 2024-04-15T15:21:36.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f520d603640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f520815cf60 0x7f52080afa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f520d603640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f520806e0b0 0x7f52080d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f520d603640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5208163090 con 0x7f52080d0f40 2024-04-15T15:21:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f520d603640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f52080d13b0 con 0x7f520815cf60 2024-04-15T15:21:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f520d603640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f52080b0160 con 0x7f520806e0b0 2024-04-15T15:21:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f5206ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f520815cf60 0x7f52080afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f5206ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f520815cf60 0x7f52080afa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46824/0 (socket says 172.21.15.12:46824) 2024-04-15T15:21:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f5206ffd640 1 -- 172.21.15.12:0/2333960462 learned_addr learned my addr 172.21.15.12:0/2333960462 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f52067fc640 1 --2- 172.21.15.12:0/2333960462 >> v2:172.21.15.55:3300/0 conn(0x7f520806e0b0 0x7f52080d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c039070 con 0x7fa3900051c0 2024-04-15T15:21:36.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7fa3a8c4b640 1 -- 172.21.15.12:0/691695333 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa39c02cca0 con 0x7fa3900051c0 2024-04-15T15:21:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7f52067fc640 1 -- 172.21.15.12:0/2333960462 >> v2:172.21.15.12:3300/0 conn(0x7f520815cf60 msgr2=0x7f52080afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7f5fc259c640 1 --2- 172.21.15.12:0/2278870485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc415a620 0x7f5fc415aa00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b1e0 tx=0x7f5fb802c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7495c62057da386 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7f52067fc640 1 --2- 172.21.15.12:0/2333960462 >> v2:172.21.15.12:3300/0 conn(0x7f520815cf60 0x7f52080afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7f52067fc640 1 -- 172.21.15.12:0/2333960462 >> v1:172.21.15.55:6789/0 conn(0x7f52080d0f40 legacy=0x7f5208166470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7fc8df59e640 1 Processor -- start 2024-04-15T15:21:36.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7f5fc159a640 1 -- 172.21.15.12:0/2278870485 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8039070 con 0x7f5fc415a620 2024-04-15T15:21:36.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f2e4493c640 1 Processor -- start 2024-04-15T15:21:36.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f5fc159a640 1 -- 172.21.15.12:0/2278870485 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb8034440 con 0x7f5fc415a620 2024-04-15T15:21:36.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7faecbb67640 1 Processor -- start 2024-04-15T15:21:36.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7faecbb67640 1 -- start start 2024-04-15T15:21:36.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f5fc159a640 1 -- 172.21.15.12:0/2278870485 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8039070 con 0x7f5fc415a620 2024-04-15T15:21:36.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7fa3abeb1640 1 -- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3900051c0 msgr2=0x7fa39002d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7fa3abeb1640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3900051c0 0x7fa39002d9d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c4c0 tx=0x7fa390005830 comp rx=0 tx=0).stop 2024-04-15T15:21:36.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f2e4493c640 1 -- start start 2024-04-15T15:21:36.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7fa3abeb1640 1 -- 172.21.15.12:0/691695333 shutdown_connections 2024-04-15T15:21:36.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7fa3abeb1640 1 --2- 172.21.15.12:0/691695333 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3900051c0 0x7fa39002d9d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7fa3abeb1640 1 -- 172.21.15.12:0/691695333 >> 172.21.15.12:0/691695333 conn(0x7fa3a4072440 msgr2=0x7fa3a4072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.726+0000 7f52067fc640 1 -- 172.21.15.12:0/2333960462 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5208163370 con 0x7f520806e0b0 2024-04-15T15:21:36.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f06bbfff640 1 Processor -- start 2024-04-15T15:21:36.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f06bbfff640 1 -- start start 2024-04-15T15:21:36.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.725+0000 7f6c4afe2640 1 -- 172.21.15.12:0/709370207 wait complete. 2024-04-15T15:21:36.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f6c4afe2640 1 Processor -- start 2024-04-15T15:21:36.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f6c4afe2640 1 -- start start 2024-04-15T15:21:36.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f52067fc640 1 --2- 172.21.15.12:0/2333960462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520806e0b0 0x7f52080d0a00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f52080d15b0 tx=0x7f51fc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baf24a3b06087301 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7faecbb67640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec40737e0 0x7faec414e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7faecbb67640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faec40cd540 0x7faec41f87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7faecbb67640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec415dd30 0x7faec4156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7faecbb67640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faec406f800 con 0x7faec40737e0 2024-04-15T15:21:36.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7faecbb67640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faec40b31e0 con 0x7faec40cd540 2024-04-15T15:21:36.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7faecbb67640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faec415e110 con 0x7faec415dd30 2024-04-15T15:21:36.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f5205ffb640 1 -- 172.21.15.12:0/2333960462 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc0315a0 con 0x7f520806e0b0 2024-04-15T15:21:36.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7f5205ffb640 1 -- 172.21.15.12:0/2333960462 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51fc037070 con 0x7f520806e0b0 2024-04-15T15:21:36.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7faec9903640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec40737e0 0x7faec414e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f5fc8a88640 1 -- 172.21.15.12:0/2278870485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc415a620 msgr2=0x7f5fc415aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/2278870485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc415a620 0x7f5fc415aa00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b1e0 tx=0x7f5fb802c180 comp rx=0 tx=0).stop 2024-04-15T15:21:36.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7faec9903640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec40737e0 0x7faec414e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46836/0 (socket says 172.21.15.12:46836) 2024-04-15T15:21:36.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7faec9903640 1 -- 172.21.15.12:0/1741916964 learned_addr learned my addr 172.21.15.12:0/1741916964 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.727+0000 7fc8df59e640 1 -- start start 2024-04-15T15:21:36.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8df59e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 0x7fc8e015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8df59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e015e680 0x7fc8e00c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8df59e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8e00c7f10 0x7fc8e01518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8df59e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e015ea60 con 0x7fc8e015e680 2024-04-15T15:21:36.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8df59e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e006d480 con 0x7fc8e00c7f10 2024-04-15T15:21:36.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8df59e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e00d2170 con 0x7fc8e00d11b0 2024-04-15T15:21:36.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f5205ffb640 1 -- 172.21.15.12:0/2333960462 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc031960 con 0x7f520806e0b0 2024-04-15T15:21:36.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e4493c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e38005ee0 0x7f2e380df000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f5fc8a88640 1 -- 172.21.15.12:0/2278870485 shutdown_connections 2024-04-15T15:21:36.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/2278870485 >> v2:172.21.15.12:3300/0 conn(0x7f5fc415b810 0x7f5fc40d0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/2278870485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc415a620 0x7f5fc415aa00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fa3abeb1640 1 -- 172.21.15.12:0/691695333 shutdown_connections 2024-04-15T15:21:36.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f5fc8a88640 1 -- 172.21.15.12:0/2278870485 >> 172.21.15.12:0/2278870485 conn(0x7f5fc406be00 msgr2=0x7f5fc406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e4493c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e38006a30 0x7f2e38035de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e4493c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e380ead00 0x7f2e38043680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e4493c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e380547c0 con 0x7f2e38005ee0 2024-04-15T15:21:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f06bbfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc15d2c0 0x7f06bc0ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f06bbfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06bc0cf820 0x7f06bc0ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f06bbfff640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06bc0cfcb0 con 0x7f06bc0cf3a0 2024-04-15T15:21:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f06bbfff640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f06bc166830 con 0x7f06bc0cf820 2024-04-15T15:21:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f06bbfff640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f06bc166b10 con 0x7f06bc15d2c0 2024-04-15T15:21:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f06bb7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06bc0cf820 0x7f06bc0ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f06bb7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06bc0cf820 0x7f06bc0ce5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46846/0 (socket says 172.21.15.12:46846) 2024-04-15T15:21:36.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f06bb7fe640 1 -- 172.21.15.12:0/3008653484 learned_addr learned my addr 172.21.15.12:0/3008653484 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e4493c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2e380501a0 con 0x7f2e38006a30 2024-04-15T15:21:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e4493c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2e38054940 con 0x7f2e380ead00 2024-04-15T15:21:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8ded9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8e00c7f10 0x7fc8e01518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8ded9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8e00c7f10 0x7fc8e01518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:37680/0 (socket says 172.21.15.12:37680) 2024-04-15T15:21:36.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8ded9d640 1 -- 172.21.15.12:0/1014953290 learned_addr learned my addr 172.21.15.12:0/1014953290 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f520d603640 1 -- 172.21.15.12:0/2333960462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520806e0b0 msgr2=0x7f52080d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f520d603640 1 --2- 172.21.15.12:0/2333960462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520806e0b0 0x7f52080d0a00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f52080d15b0 tx=0x7f51fc02b840 comp rx=0 tx=0).stop 2024-04-15T15:21:36.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e3f59e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e380ead00 0x7f2e38043680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e3f59e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e380ead00 0x7f2e38043680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33878/0 (socket says 172.21.15.12:33878) 2024-04-15T15:21:36.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f2e3f59e640 1 -- 172.21.15.12:0/414183314 learned_addr learned my addr 172.21.15.12:0/414183314 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f520d603640 1 -- 172.21.15.12:0/2333960462 shutdown_connections 2024-04-15T15:21:36.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f520d603640 1 --2- 172.21.15.12:0/2333960462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520806e0b0 0x7f52080d0a00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f520d603640 1 --2- 172.21.15.12:0/2333960462 >> v2:172.21.15.12:3300/0 conn(0x7f520815cf60 0x7f52080afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7f520d603640 1 -- 172.21.15.12:0/2333960462 >> 172.21.15.12:0/2333960462 conn(0x7f520806be00 msgr2=0x7f520806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7fc8d7fff640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e015e680 0x7fc8e00c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fc8de59c640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 0x7fc8e015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.728+0000 7fa3abeb1640 1 -- 172.21.15.12:0/691695333 wait complete. 2024-04-15T15:21:36.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f06baffd640 1 --2- 172.21.15.12:0/3008653484 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc15d2c0 0x7f06bc0ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4afe2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c440b36b0 0x7f6c44168580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4afe2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 0x7f6c440cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7faec9903640 1 -- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec415dd30 msgr2=0x7faec4156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7faec9903640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec415dd30 0x7faec4156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7faec9903640 1 -- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faec40cd540 msgr2=0x7faec41f87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4afe2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c4416ef60 0x7f6c440cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7faec9102640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faec40cd540 0x7faec41f87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f2e3f59e640 1 -- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e38006a30 msgr2=0x7f2e38035de0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:21:36.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4afe2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6c440b33e0 con 0x7f6c440b36b0 2024-04-15T15:21:36.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4afe2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6c440b3260 con 0x7f6c4416ef60 2024-04-15T15:21:36.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f5fc8a88640 1 -- 172.21.15.12:0/2278870485 shutdown_connections 2024-04-15T15:21:36.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f2e3f59e640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e38006a30 0x7f2e38035de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4afe2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6c440b3b40 con 0x7f6c4416d1f0 2024-04-15T15:21:36.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f2e3f59e640 1 -- 172.21.15.12:0/414183314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e38005ee0 msgr2=0x7f2e380df000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f2e3f59e640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e38005ee0 0x7f2e380df000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4957f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c4416ef60 0x7f6c440cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4957f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c4416ef60 0x7f6c440cffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:37686/0 (socket says 172.21.15.12:37686) 2024-04-15T15:21:36.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f6c4957f640 1 -- 172.21.15.12:0/2640781118 learned_addr learned my addr 172.21.15.12:0/2640781118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f5fc8a88640 1 -- 172.21.15.12:0/2278870485 wait complete. 2024-04-15T15:21:36.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7faec9903640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faec40cd540 0x7faec41f87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7faec9903640 1 -- 172.21.15.12:0/1741916964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec41f9240 con 0x7faec40737e0 2024-04-15T15:21:36.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f2e3f59e640 1 -- 172.21.15.12:0/414183314 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e380df540 con 0x7f2e380ead00 2024-04-15T15:21:36.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f520d603640 1 -- 172.21.15.12:0/2333960462 shutdown_connections 2024-04-15T15:21:36.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f2e3f59e640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e380ead00 0x7f2e38043680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e300098c0 tx=0x7f2e300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7faec9903640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec40737e0 0x7faec414e3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7faebc03a040 tx=0x7faebc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f520d603640 1 -- 172.21.15.12:0/2333960462 wait complete. 2024-04-15T15:21:36.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7fc8ded9d640 1 -- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 msgr2=0x7fc8e015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7fc8ded9d640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 0x7fc8e015ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7fc8ded9d640 1 -- 172.21.15.12:0/1014953290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e015e680 msgr2=0x7fc8e00c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7fc8ded9d640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e015e680 0x7fc8e00c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7fc8ded9d640 1 -- 172.21.15.12:0/1014953290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e00c9140 con 0x7fc8e00c7f10 2024-04-15T15:21:36.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7fc8ded9d640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8e00c7f10 0x7fc8e01518e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800b9c0 tx=0x7fc8d800be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f2e277fe640 1 -- 172.21.15.12:0/414183314 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30048070 con 0x7f2e380ead00 2024-04-15T15:21:36.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7fa3abeb1640 1 Processor -- start 2024-04-15T15:21:36.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f2e277fe640 1 -- 172.21.15.12:0/414183314 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e300042d0 con 0x7f2e380ead00 2024-04-15T15:21:36.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7fc8d6ffd640 1 -- 172.21.15.12:0/1014953290 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8019070 con 0x7fc8e00c7f10 2024-04-15T15:21:36.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e38034420 con 0x7f2e380ead00 2024-04-15T15:21:36.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7faeba7fc640 1 -- 172.21.15.12:0/1741916964 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc038400 con 0x7faec40737e0 2024-04-15T15:21:36.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7faecbb67640 1 -- 172.21.15.12:0/1741916964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec41f9520 con 0x7faec40737e0 2024-04-15T15:21:36.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e380472c0 con 0x7f2e380ead00 2024-04-15T15:21:36.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7faecbb67640 1 -- 172.21.15.12:0/1741916964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec4072670 con 0x7faec40737e0 2024-04-15T15:21:36.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f6c4957f640 1 -- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 msgr2=0x7f6c440cfa70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:21:36.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f6c4957f640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 0x7f6c440cfa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f6c4957f640 1 -- 172.21.15.12:0/2640781118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c440b36b0 msgr2=0x7f6c44168580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f6c4957f640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c440b36b0 0x7f6c44168580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7f6c4957f640 1 -- 172.21.15.12:0/2640781118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c4409de30 con 0x7f6c4416ef60 2024-04-15T15:21:36.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7fa3abeb1640 1 -- start start 2024-04-15T15:21:36.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7f5fc8a88640 1 Processor -- start 2024-04-15T15:21:36.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7fa3abeb1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3900051c0 0x7fa3a40cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7fa3abeb1640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3a414dae0 0x7fa3a414eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7fa3abeb1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3a414f040 0x7fa3a4157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3a414f040 0x7fa3a4157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3a414f040 0x7fa3a4157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46882/0 (socket says 172.21.15.12:46882) 2024-04-15T15:21:36.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7f520d603640 1 Processor -- start 2024-04-15T15:21:36.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f520d603640 1 -- start start 2024-04-15T15:21:36.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7faeaffff640 1 -- 172.21.15.12:0/1741916964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea4005290 con 0x7faec40737e0 2024-04-15T15:21:36.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3a944c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3a414dae0 0x7fa3a414eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3abeb1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a40b3e80 con 0x7fa3a414f040 2024-04-15T15:21:36.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3abeb1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a40b3ba0 con 0x7fa3900051c0 2024-04-15T15:21:36.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3abeb1640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a40d0430 con 0x7fa3a414dae0 2024-04-15T15:21:36.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 -- 172.21.15.12:0/2998343689 learned_addr learned my addr 172.21.15.12:0/2998343689 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 -- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3a414dae0 msgr2=0x7fa3a414eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3a414dae0 0x7fa3a414eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 -- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3900051c0 msgr2=0x7fa3a40cfbc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:21:36.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3900051c0 0x7fa3a40cfbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 -- 172.21.15.12:0/2998343689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a40cee00 con 0x7fa3a414f040 2024-04-15T15:21:36.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7faeba7fc640 1 -- 172.21.15.12:0/1741916964 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faebc008040 con 0x7faec40737e0 2024-04-15T15:21:36.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f6c4957f640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c4416ef60 0x7f6c440cffb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c7a0 tx=0x7f6c3c00cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e00c9410 con 0x7fc8e00c7f10 2024-04-15T15:21:36.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.730+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e00b85e0 con 0x7fc8e00c7f10 2024-04-15T15:21:36.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7fc8d6ffd640 1 -- 172.21.15.12:0/1014953290 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d800cba0 con 0x7fc8e00c7f10 2024-04-15T15:21:36.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fc8d6ffd640 1 -- 172.21.15.12:0/1014953290 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8014440 con 0x7fc8e00c7f10 2024-04-15T15:21:36.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f06bb7fe640 1 -- 172.21.15.12:0/3008653484 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc15d2c0 msgr2=0x7f06bc0ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:36.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f06bb7fe640 1 --2- 172.21.15.12:0/3008653484 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc15d2c0 0x7f06bc0ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f06bb7fe640 1 -- 172.21.15.12:0/3008653484 >> v1:172.21.15.12:6789/0 conn(0x7f06bc0cf3a0 legacy=0x7f06bc0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.729+0000 7f06bb7fe640 1 -- 172.21.15.12:0/3008653484 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06bc166df0 con 0x7f06bc0cf820 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7f06bb7fe640 1 --2- 172.21.15.12:0/3008653484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc0cf820 0x7f06bc0ce5d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009e50 tx=0x7f06b402bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de92d6a07a49a0b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7f06b9ffb640 1 -- 172.21.15.12:0/3008653484 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b402caa0 con 0x7f06bc0cf820 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7f06b9ffb640 1 -- 172.21.15.12:0/3008653484 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b402cc40 con 0x7f06bc0cf820 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.731+0000 7f06b9ffb640 1 -- 172.21.15.12:0/3008653484 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b402caa0 con 0x7f06bc0cf820 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f06bbfff640 1 -- 172.21.15.12:0/3008653484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc0cf820 msgr2=0x7f06bc0ce5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f06bbfff640 1 --2- 172.21.15.12:0/3008653484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc0cf820 0x7f06bc0ce5d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009e50 tx=0x7f06b402bf00 comp rx=0 tx=0).stop 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f06bbfff640 1 -- 172.21.15.12:0/3008653484 shutdown_connections 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f06bbfff640 1 --2- 172.21.15.12:0/3008653484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc0cf820 0x7f06bc0ce5d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f06bbfff640 1 --2- 172.21.15.12:0/3008653484 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc15d2c0 0x7f06bc0ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f06bbfff640 1 -- 172.21.15.12:0/3008653484 >> 172.21.15.12:0/3008653484 conn(0x7f06bc0b1c50 msgr2=0x7f06bc0b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:36.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7fa3aa44e640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3a414f040 0x7fa3a4157540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa39c034d70 tx=0x7fa39c001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f06bbfff640 1 -- 172.21.15.12:0/3008653484 shutdown_connections 2024-04-15T15:21:36.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7faeba7fc640 1 -- 172.21.15.12:0/1741916964 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc008430 con 0x7faec40737e0 2024-04-15T15:21:36.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f520d603640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f520806e0b0 0x7f5208151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f520d603640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52080d0f40 0x7f520814cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f520d603640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520815cf60 0x7f5208165230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f520d603640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5208163930 con 0x7f52080d0f40 2024-04-15T15:21:36.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f520d603640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f52080b5bf0 con 0x7f520815cf60 2024-04-15T15:21:36.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f2e277fe640 1 -- 172.21.15.12:0/414183314 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30003610 con 0x7f2e380ead00 2024-04-15T15:21:36.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f6c417fa640 1 -- 172.21.15.12:0/2640781118 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c003b70 con 0x7f6c4416ef60 2024-04-15T15:21:36.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f520d603640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f52080720b0 con 0x7f520806e0b0 2024-04-15T15:21:36.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e20005290 con 0x7f2e380ead00 2024-04-15T15:21:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f6c417fa640 1 -- 172.21.15.12:0/2640781118 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3c003d10 con 0x7f6c4416ef60 2024-04-15T15:21:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.732+0000 7f5fc8a88640 1 -- start start 2024-04-15T15:21:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc8a88640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc40cff20 0x7f5fc4155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc8a88640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fc415a620 0x7f5fc41e42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc8a88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc415b810 0x7f5fc41e6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc8a88640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc40689e0 con 0x7f5fc415a620 2024-04-15T15:21:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc8a88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc415b2f0 con 0x7f5fc415b810 2024-04-15T15:21:36.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc8a88640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5fc4068b60 con 0x7f5fc40cff20 2024-04-15T15:21:36.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c4005290 con 0x7fc8e00c7f10 2024-04-15T15:21:36.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7fa39a7fc640 1 -- 172.21.15.12:0/2998343689 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c045070 con 0x7fa3a414f040 2024-04-15T15:21:36.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7fa39a7fc640 1 -- 172.21.15.12:0/2998343689 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa39c02ce50 con 0x7fa3a414f040 2024-04-15T15:21:36.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a40cf0e0 con 0x7fa3a414f040 2024-04-15T15:21:36.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fc415a620 0x7f5fc41e42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fc415a620 0x7f5fc41e42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46898/0 (socket says 172.21.15.12:46898) 2024-04-15T15:21:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5fc1d9b640 1 -- 172.21.15.12:0/1550637082 learned_addr learned my addr 172.21.15.12:0/1550637082 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f2e277fe640 1 -- 172.21.15.12:0/414183314 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e30038020 con 0x7f2e380ead00 2024-04-15T15:21:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f2e277fe640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2e2c06c9c0 0x7f2e2c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7f2e277fe640 1 -- 172.21.15.12:0/414183314 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2e300ad1b0 con 0x7f2e380ead00 2024-04-15T15:21:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7fa39a7fc640 1 -- 172.21.15.12:0/2998343689 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c039040 con 0x7fa3a414f040 2024-04-15T15:21:36.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a414f840 con 0x7fa3a414f040 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f5207fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520815cf60 0x7f5208165230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5207fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520815cf60 0x7f5208165230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:37708/0 (socket says 172.21.15.12:37708) 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5207fff640 1 -- 172.21.15.12:0/695624473 learned_addr learned my addr 172.21.15.12:0/695624473 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f52067fc640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52080d0f40 0x7f520814cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5207fff640 1 -- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f520806e0b0 msgr2=0x7f5208151630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5207fff640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f520806e0b0 0x7f5208151630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5207fff640 1 -- 172.21.15.12:0/695624473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52080d0f40 msgr2=0x7f520814cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5207fff640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52080d0f40 0x7f520814cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5207fff640 1 -- 172.21.15.12:0/695624473 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5208164190 con 0x7f520815cf60 2024-04-15T15:21:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5fc1d9b640 1 -- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc40cff20 msgr2=0x7f5fc4155360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:21:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5fc1d9b640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc40cff20 0x7f5fc4155360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5fc1d9b640 1 -- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc415b810 msgr2=0x7f5fc41e6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5fc1d9b640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc415b810 0x7f5fc41e6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.734+0000 7f5fc1d9b640 1 -- 172.21.15.12:0/1550637082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc41e7510 con 0x7f5fc415a620 2024-04-15T15:21:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7f2e3ed9d640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2e2c06c9c0 0x7f2e2c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7f5fc1d9b640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fc415a620 0x7f5fc41e42e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400b4a0 tx=0x7f5fb400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7f2e3ed9d640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2e2c06c9c0 0x7f2e2c06ee80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2e4006aee0 tx=0x7f2e4006ddd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7f5fb2ffd640 1 -- 172.21.15.12:0/1550637082 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4014070 con 0x7f5fc415a620 2024-04-15T15:21:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7f5fb2ffd640 1 -- 172.21.15.12:0/1550637082 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb4010610 con 0x7f5fc415a620 2024-04-15T15:21:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7faeba7fc640 1 -- 172.21.15.12:0/1741916964 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faebc047030 con 0x7faec40737e0 2024-04-15T15:21:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a40690f0 con 0x7fa3a414f040 2024-04-15T15:21:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7fc8d6ffd640 1 -- 172.21.15.12:0/1014953290 <== mon.1 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc8d801c070 con 0x7fc8e00c7f10 2024-04-15T15:21:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f5fb2ffd640 1 -- 172.21.15.12:0/1550637082 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb40109e0 con 0x7f5fc415a620 2024-04-15T15:21:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc4157ab0 con 0x7f5fc415a620 2024-04-15T15:21:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc4158070 con 0x7f5fc415a620 2024-04-15T15:21:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7faeba7fc640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faea806c970 0x7faea806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.733+0000 7f06bbfff640 1 -- 172.21.15.12:0/3008653484 wait complete. 2024-04-15T15:21:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c4409e110 con 0x7f6c4416ef60 2024-04-15T15:21:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7faeba7fc640 1 -- 172.21.15.12:0/1741916964 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faebc0ad9e0 con 0x7faec40737e0 2024-04-15T15:21:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f6c417fa640 1 -- 172.21.15.12:0/2640781118 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c00aa90 con 0x7f6c4416ef60 2024-04-15T15:21:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c44214060 con 0x7f6c4416ef60 2024-04-15T15:21:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7fc8d6ffd640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8c006c940 0x7fc8c006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7faec9102640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faea806c970 0x7faea806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7fc8de59c640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8c006c940 0x7fc8c006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7f5207fff640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520815cf60 0x7f5208165230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f51f800c3c0 tx=0x7f51f800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7fc8d6ffd640 1 -- 172.21.15.12:0/1014953290 <== mon.1 v2:172.21.15.55:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc8d808bed0 con 0x7fc8e00c7f10 2024-04-15T15:21:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7fc8de59c640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8c006c940 0x7fc8c006ee00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0000f60 tx=0x7fc8d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7faec9102640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faea806c970 0x7faea806ee30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faec409aff0 tx=0x7faec0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7f06bbfff640 1 Processor -- start 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.735+0000 7f06bbfff640 1 -- start start 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bbfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc0cf820 0x7f06bc08b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06bc15d2c0 0x7f06bc208de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc20cc00 0x7f06bc20d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bbfff640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc0acf30 con 0x7f06bc20cc00 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bbfff640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc0cd440 con 0x7f06bc15d2c0 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bbfff640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f06bc0ad0b0 con 0x7f06bc0cf820 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc20cc00 0x7f06bc20d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc20cc00 0x7f06bc20d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46914/0 (socket says 172.21.15.12:46914) 2024-04-15T15:21:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7f06bb7fe640 1 -- 172.21.15.12:0/2375517788 learned_addr learned my addr 172.21.15.12:0/2375517788 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7f06bb7fe640 1 -- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc0cf820 msgr2=0x7f06bc08b940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:21:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7f06bb7fe640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc0cf820 0x7f06bc08b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7f06bb7fe640 1 -- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06bc15d2c0 msgr2=0x7f06bc208de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:21:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7f06bb7fe640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06bc15d2c0 0x7f06bc208de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7f06bb7fe640 1 -- 172.21.15.12:0/2375517788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06bc1610c0 con 0x7f06bc20cc00 2024-04-15T15:21:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.737+0000 7f06bb7fe640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc20cc00 0x7f06bc20d090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009f70 tx=0x7f06b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.736+0000 7fa39a7fc640 1 -- 172.21.15.12:0/2998343689 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa39c048070 con 0x7fa3a414f040 2024-04-15T15:21:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.738+0000 7fa39a7fc640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa38806c9f0 0x7fa38806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.738+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c440cc8e0 con 0x7f6c4416ef60 2024-04-15T15:21:36.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.738+0000 7fa39a7fc640 1 -- 172.21.15.12:0/2998343689 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa39c0acee0 con 0x7fa3a414f040 2024-04-15T15:21:36.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.738+0000 7fa3a9c4d640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa38806c9f0 0x7fa38806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.738+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc40690f0 con 0x7f5fc415a620 2024-04-15T15:21:36.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.738+0000 7fa3a9c4d640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa38806c9f0 0x7fa38806eeb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa3a406a7b0 tx=0x7fa3a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.739+0000 7f51ef7fe640 1 -- 172.21.15.12:0/695624473 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f80052d0 con 0x7f520815cf60 2024-04-15T15:21:36.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.739+0000 7f2e277fe640 1 -- 172.21.15.12:0/414183314 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e380df540 con 0x7f2e380ead00 2024-04-15T15:21:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.740+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5208164470 con 0x7f520815cf60 2024-04-15T15:21:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.740+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5208164a30 con 0x7f520815cf60 2024-04-15T15:21:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.740+0000 7f51ef7fe640 1 -- 172.21.15.12:0/695624473 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f8005470 con 0x7f520815cf60 2024-04-15T15:21:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.740+0000 7fc8d6ffd640 1 -- 172.21.15.12:0/1014953290 <== mon.1 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8d8023020 con 0x7fc8e00c7f10 2024-04-15T15:21:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.740+0000 7f51ef7fe640 1 -- 172.21.15.12:0/695624473 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8020060 con 0x7f520815cf60 2024-04-15T15:21:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.741+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52080753d0 con 0x7f520815cf60 2024-04-15T15:21:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.741+0000 7f6c417fa640 1 -- 172.21.15.12:0/2640781118 <== mon.1 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c3c00ac30 con 0x7f6c4416ef60 2024-04-15T15:21:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.741+0000 7f51ef7fe640 1 -- 172.21.15.12:0/695624473 <== mon.1 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f51f8007220 con 0x7f520815cf60 2024-04-15T15:21:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.742+0000 7f51ef7fe640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51f406c8f0 0x7f51f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.742+0000 7f6c417fa640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6c3006c940 0x7f6c3006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.742+0000 7f6c417fa640 1 -- 172.21.15.12:0/2640781118 <== mon.1 v2:172.21.15.55:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6c3c08c950 con 0x7f6c4416ef60 2024-04-15T15:21:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.742+0000 7f5206ffd640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51f406c8f0 0x7f51f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.742+0000 7f6c48d7e640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6c3006c940 0x7f6c3006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.743+0000 7f5206ffd640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51f406c8f0 0x7f51f406edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5200009400 tx=0x7f5200007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.743+0000 7f6c48d7e640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6c3006c940 0x7f6c3006ee00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009690 tx=0x7f6c38006bd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.743+0000 7faeba7fc640 1 -- 172.21.15.12:0/1741916964 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faebc04e060 con 0x7faec40737e0 2024-04-15T15:21:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.745+0000 7f51ef7fe640 1 -- 172.21.15.12:0/695624473 <== mon.1 v2:172.21.15.55:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f51f8056d80 con 0x7f520815cf60 2024-04-15T15:21:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.745+0000 7f6c417fa640 1 -- 172.21.15.12:0/2640781118 <== mon.1 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6c440cc8e0 con 0x7f6c4416ef60 2024-04-15T15:21:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.750+0000 7f51ef7fe640 1 -- 172.21.15.12:0/695624473 <== mon.1 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52080753d0 con 0x7f520815cf60 2024-04-15T15:21:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.753+0000 7fa39a7fc640 1 -- 172.21.15.12:0/2998343689 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa39c07a960 con 0x7fa3a414f040 2024-04-15T15:21:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.753+0000 7f06b37fe640 1 -- 172.21.15.12:0/2375517788 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4042020 con 0x7f06bc20cc00 2024-04-15T15:21:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.753+0000 7f06b37fe640 1 -- 172.21.15.12:0/2375517788 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b40094d0 con 0x7f06bc20cc00 2024-04-15T15:21:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.754+0000 7f06b37fe640 1 -- 172.21.15.12:0/2375517788 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b40045f0 con 0x7f06bc20cc00 2024-04-15T15:21:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.755+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06bc15a310 con 0x7f06bc20cc00 2024-04-15T15:21:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.755+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06bc15a8d0 con 0x7f06bc20cc00 2024-04-15T15:21:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.754+0000 7f5fb2ffd640 1 -- 172.21.15.12:0/1550637082 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5fb4017070 con 0x7f5fc415a620 2024-04-15T15:21:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.755+0000 7f5fb2ffd640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5fa406c9c0 0x7f5fa406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.755+0000 7f5fb2ffd640 1 -- 172.21.15.12:0/1550637082 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5fb408bd80 con 0x7f5fc415a620 2024-04-15T15:21:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.757+0000 7f5fc259c640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5fa406c9c0 0x7f5fa406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.757+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06bc04fd50 con 0x7f06bc20cc00 2024-04-15T15:21:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.758+0000 7f5fc259c640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5fa406c9c0 0x7f5fa406ee80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8036000 tx=0x7f5fb8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.763+0000 7f5fb2ffd640 1 -- 172.21.15.12:0/1550637082 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5fb4059ee0 con 0x7f5fc415a620 2024-04-15T15:21:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.766+0000 7f06b37fe640 1 -- 172.21.15.12:0/2375517788 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f06b4045020 con 0x7f06bc20cc00 2024-04-15T15:21:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.766+0000 7f06b37fe640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f069c06c9c0 0x7f069c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.766+0000 7f06b37fe640 1 -- 172.21.15.12:0/2375517788 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f06b40ae340 con 0x7f06bc20cc00 2024-04-15T15:21:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.767+0000 7f06baffd640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f069c06c9c0 0x7f069c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.774+0000 7f06b37fe640 1 -- 172.21.15.12:0/2375517788 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06b404c030 con 0x7f06bc20cc00 2024-04-15T15:21:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:36.779+0000 7f06baffd640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f069c06c9c0 0x7f069c06ee80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009930 tx=0x7f06ac009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:37.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.168+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc8c4005850 con 0x7fc8e00c7f10 2024-04-15T15:21:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.169+0000 7fc8d6ffd640 1 -- 172.21.15.12:0/1014953290 <== mon.1 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc8d8059ae0 con 0x7fc8e00c7f10 2024-04-15T15:21:37.171 INFO:teuthology.orchestra.run.smithi012.stdout:55834574858 2024-04-15T15:21:37.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.172+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f52080690f0 con 0x7f520815cf60 2024-04-15T15:21:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8c006c940 msgr2=0x7fc8c006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8c006c940 0x7fc8c006ee00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0000f60 tx=0x7fc8d0009040 comp rx=0 tx=0).stop 2024-04-15T15:21:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8e00c7f10 msgr2=0x7fc8e01518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8e00c7f10 0x7fc8e01518e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800b9c0 tx=0x7fc8d800be80 comp rx=0 tx=0).stop 2024-04-15T15:21:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 shutdown_connections 2024-04-15T15:21:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8c006c940 0x7fc8c006ee00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8e00c7f10 0x7fc8e01518e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8e015e680 0x7fc8e00c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 --2- 172.21.15.12:0/1014953290 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8e00d11b0 0x7fc8e015ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 >> 172.21.15.12:0/1014953290 conn(0x7fc8e009d740 msgr2=0x7fc8e00b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.173+0000 7f51ef7fe640 1 -- 172.21.15.12:0/695624473 <== mon.1 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f51f8058ce0 con 0x7f520815cf60 2024-04-15T15:21:37.176 INFO:teuthology.orchestra.run.smithi012.stdout:55834574860 2024-04-15T15:21:37.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.175+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f6c440689e0 con 0x7f6c4416ef60 2024-04-15T15:21:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.175+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 shutdown_connections 2024-04-15T15:21:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.175+0000 7fc8df59e640 1 -- 172.21.15.12:0/1014953290 wait complete. 2024-04-15T15:21:37.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.175+0000 7f6c417fa640 1 -- 172.21.15.12:0/2640781118 <== mon.1 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6c3c05aa20 con 0x7f6c4416ef60 2024-04-15T15:21:37.178 INFO:teuthology.orchestra.run.smithi012.stdout:55834574859 2024-04-15T15:21:37.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.176+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2e20005580 con 0x7f2e380ead00 2024-04-15T15:21:37.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.176+0000 7faeaffff640 1 -- 172.21.15.12:0/1741916964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7faea4005850 con 0x7faec40737e0 2024-04-15T15:21:37.178 INFO:teuthology.orchestra.run.smithi012.stdout:55834574858 2024-04-15T15:21:37.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.176+0000 7f2e277fe640 1 -- 172.21.15.12:0/414183314 <== mon.2 v2:172.21.15.55:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2e3007b3a0 con 0x7f2e380ead00 2024-04-15T15:21:37.179 INFO:teuthology.orchestra.run.smithi012.stdout:55834574858 2024-04-15T15:21:37.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.176+0000 7faeba7fc640 1 -- 172.21.15.12:0/1741916964 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faebc07bc40 con 0x7faec40737e0 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.177+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51f406c8f0 msgr2=0x7f51f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.177+0000 7f520d603640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51f406c8f0 0x7f51f406edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5200009400 tx=0x7f5200007040 comp rx=0 tx=0).stop 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.177+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520815cf60 msgr2=0x7f5208165230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.177+0000 7f520d603640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520815cf60 0x7f5208165230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f51f800c3c0 tx=0x7f51f800c880 comp rx=0 tx=0).stop 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.178+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 shutdown_connections 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.178+0000 7f520d603640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51f406c8f0 0x7f51f406edb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.178+0000 7f520d603640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f520815cf60 0x7f5208165230 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.178+0000 7f520d603640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52080d0f40 0x7f520814cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.178+0000 7f520d603640 1 --2- 172.21.15.12:0/695624473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f520806e0b0 0x7f5208151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.178+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 >> 172.21.15.12:0/695624473 conn(0x7f520806be00 msgr2=0x7f5208072320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.178+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6c3006c940 msgr2=0x7f6c3006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.178+0000 7f6c4afe2640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6c3006c940 0x7f6c3006ee00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009690 tx=0x7f6c38006bd0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c4416ef60 msgr2=0x7f6c440cffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c4416ef60 0x7f6c440cffb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c7a0 tx=0x7f6c3c00cc60 comp rx=0 tx=0).stop 2024-04-15T15:21:37.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 shutdown_connections 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6c3006c940 0x7f6c3006ee00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c4416ef60 0x7f6c440cffb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6c4416d1f0 0x7f6c440cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 --2- 172.21.15.12:0/2640781118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c440b36b0 0x7f6c44168580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 >> 172.21.15.12:0/2640781118 conn(0x7f6c440b5680 msgr2=0x7f6c4416bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 shutdown_connections 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f6c4afe2640 1 -- 172.21.15.12:0/2640781118 wait complete. 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2e2c06c9c0 msgr2=0x7f2e2c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f2e4493c640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2e2c06c9c0 0x7f2e2c06ee80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2e4006aee0 tx=0x7f2e4006ddd0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e380ead00 msgr2=0x7f2e38043680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e380ead00 0x7f2e38043680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e300098c0 tx=0x7f2e300046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 shutdown_connections 2024-04-15T15:21:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 -- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faea806c970 msgr2=0x7faea806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faea806c970 0x7faea806ee30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faec409aff0 tx=0x7faec0009040 comp rx=0 tx=0).stop 2024-04-15T15:21:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 -- 172.21.15.12:0/1741916964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec40737e0 msgr2=0x7faec414e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec40737e0 0x7faec414e3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7faebc03a040 tx=0x7faebc004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 -- 172.21.15.12:0/1741916964 shutdown_connections 2024-04-15T15:21:37.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.179+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 shutdown_connections 2024-04-15T15:21:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f520d603640 1 -- 172.21.15.12:0/695624473 wait complete. 2024-04-15T15:21:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e380ead00 0x7f2e38043680 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2e2c06c9c0 0x7f2e2c06ee80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faec415dd30 0x7faec4156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e38006a30 0x7f2e38035de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 --2- 172.21.15.12:0/414183314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e38005ee0 0x7f2e380df000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 >> 172.21.15.12:0/414183314 conn(0x7f2e38009fb0 msgr2=0x7f2e380056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faea806c970 0x7faea806ee30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faec40cd540 0x7faec41f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 --2- 172.21.15.12:0/1741916964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faec40737e0 0x7faec414e3c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 -- 172.21.15.12:0/1741916964 >> 172.21.15.12:0/1741916964 conn(0x7faec40b1c50 msgr2=0x7faec415c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 shutdown_connections 2024-04-15T15:21:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7f2e4493c640 1 -- 172.21.15.12:0/414183314 wait complete. 2024-04-15T15:21:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 -- 172.21.15.12:0/1741916964 shutdown_connections 2024-04-15T15:21:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.180+0000 7faeaffff640 1 -- 172.21.15.12:0/1741916964 wait complete. 2024-04-15T15:21:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.185+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa3a4072f70 con 0x7fa3a414f040 2024-04-15T15:21:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.186+0000 7fa39a7fc640 1 -- 172.21.15.12:0/2998343689 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa39c07ec60 con 0x7fa3a414f040 2024-04-15T15:21:37.188 INFO:teuthology.orchestra.run.smithi012.stdout:55834574859 2024-04-15T15:21:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa38806c9f0 msgr2=0x7fa38806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa38806c9f0 0x7fa38806eeb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa3a406a7b0 tx=0x7fa3a0008040 comp rx=0 tx=0).stop 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3a414f040 msgr2=0x7fa3a4157540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3a414f040 0x7fa3a4157540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa39c034d70 tx=0x7fa39c001cf0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 shutdown_connections 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3a414f040 0x7fa3a4157540 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa3a414dae0 0x7fa3a414eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa38806c9f0 0x7fa38806eeb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 --2- 172.21.15.12:0/2998343689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3900051c0 0x7fa3a40cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 >> 172.21.15.12:0/2998343689 conn(0x7fa3a4072440 msgr2=0x7fa3a4156490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5fc40b0d10 con 0x7f5fc415a620 2024-04-15T15:21:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7f5fb2ffd640 1 -- 172.21.15.12:0/1550637082 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5fb40597e0 con 0x7f5fc415a620 2024-04-15T15:21:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 shutdown_connections 2024-04-15T15:21:37.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.193+0000 7fa3abeb1640 1 -- 172.21.15.12:0/2998343689 wait complete. 2024-04-15T15:21:37.196 INFO:teuthology.orchestra.run.smithi012.stdout:55834574859 2024-04-15T15:21:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.198+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5fa406c9c0 msgr2=0x7f5fa406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.198+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5fa406c9c0 0x7f5fa406ee80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8036000 tx=0x7f5fb8030000 comp rx=0 tx=0).stop 2024-04-15T15:21:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fc415a620 msgr2=0x7f5fc41e42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fc415a620 0x7f5fc41e42e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400b4a0 tx=0x7f5fb400b960 comp rx=0 tx=0).stop 2024-04-15T15:21:37.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 shutdown_connections 2024-04-15T15:21:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5fa406c9c0 0x7f5fa406ee80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc415b810 0x7f5fc41e6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fc415a620 0x7f5fc41e42e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 --2- 172.21.15.12:0/1550637082 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fc40cff20 0x7f5fc4155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 >> 172.21.15.12:0/1550637082 conn(0x7f5fc406be00 msgr2=0x7f5fc4157130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 shutdown_connections 2024-04-15T15:21:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f5fc8a88640 1 -- 172.21.15.12:0/1550637082 wait complete. 2024-04-15T15:21:37.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.199+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f06bc0adaf0 con 0x7f06bc20cc00 2024-04-15T15:21:37.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.200+0000 7f06b37fe640 1 -- 172.21.15.12:0/2375517788 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f06b407c5c0 con 0x7f06bc20cc00 2024-04-15T15:21:37.202 INFO:teuthology.orchestra.run.smithi012.stdout:55834574858 2024-04-15T15:21:37.203 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.5 2024-04-15T15:21:37.203 DEBUG:teuthology.parallel:result is None 2024-04-15T15:21:37.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-04-15T15:21:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.202+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f069c06c9c0 msgr2=0x7f069c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.202+0000 7f06bbfff640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f069c06c9c0 0x7f069c06ee80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009930 tx=0x7f06ac009190 comp rx=0 tx=0).stop 2024-04-15T15:21:37.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc20cc00 msgr2=0x7f06bc20d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc20cc00 0x7f06bc20d090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009f70 tx=0x7f06b4004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:37.205 DEBUG:teuthology.parallel:result is None 2024-04-15T15:21:37.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.6 2024-04-15T15:21:37.206 DEBUG:teuthology.parallel:result is None 2024-04-15T15:21:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 shutdown_connections 2024-04-15T15:21:37.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06bc20cc00 0x7f06bc20d090 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f069c06c9c0 0x7f069c06ee80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06bc15d2c0 0x7f06bc208de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 --2- 172.21.15.12:0/2375517788 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06bc0cf820 0x7f06bc08b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 >> 172.21.15.12:0/2375517788 conn(0x7f06bc0b1c50 msgr2=0x7f06bc15b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 shutdown_connections 2024-04-15T15:21:37.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.203+0000 7f06bbfff640 1 -- 172.21.15.12:0/2375517788 wait complete. 2024-04-15T15:21:37.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2024-04-15T15:21:37.208 DEBUG:teuthology.parallel:result is None 2024-04-15T15:21:37.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.2 2024-04-15T15:21:37.209 DEBUG:teuthology.parallel:result is None 2024-04-15T15:21:37.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-04-15T15:21:37.214 DEBUG:teuthology.parallel:result is None 2024-04-15T15:21:37.215 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.7 2024-04-15T15:21:37.215 DEBUG:teuthology.parallel:result is None 2024-04-15T15:21:37.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2024-04-15T15:21:37.219 DEBUG:teuthology.parallel:result is None 2024-04-15T15:21:37.219 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-15T15:21:37.219 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.362+0000 7f3b07c7b640 1 Processor -- start 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.362+0000 7f3b07c7b640 1 -- start start 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b07c7b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b000d5150 0x7f3b000d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b07c7b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0015eb70 con 0x7f3b0015e790 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b07c7b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b000d3c80 con 0x7f3b000d15e0 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b07c7b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3b000d3790 con 0x7f3b000d5150 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b06218640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b000d5150 0x7f3b000d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b05a17640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3b0015e790 0x7f3b000b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56402/0 (socket says 172.21.15.12:56402) 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b05a17640 1 -- 172.21.15.12:0/3819481297 learned_addr learned my addr 172.21.15.12:0/3819481297 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b06218640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3b000d5150 0x7f3b000d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46922/0 (socket says 172.21.15.12:46922) 2024-04-15T15:21:37.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b06218640 1 -- 172.21.15.12:0/3819481297 >> v1:172.21.15.55:6789/0 conn(0x7f3b000d15e0 legacy=0x7f3b000d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b06218640 1 -- 172.21.15.12:0/3819481297 >> v1:172.21.15.12:6789/0 conn(0x7f3b0015e790 legacy=0x7f3b000b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.363+0000 7f3b06218640 1 -- 172.21.15.12:0/3819481297 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b000d3f60 con 0x7f3b000d5150 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.364+0000 7f3b06218640 1 --2- 172.21.15.12:0/3819481297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000d1a90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3b000b61c0 tx=0x7f3af802b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4cc87ed8ce53c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.364+0000 7f3b04a15640 1 -- 172.21.15.12:0/3819481297 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af8039070 con 0x7f3b000d5150 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.364+0000 7f3b04a15640 1 -- 172.21.15.12:0/3819481297 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af8030070 con 0x7f3b000d5150 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.364+0000 7f3b04a15640 1 -- 172.21.15.12:0/3819481297 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80318f0 con 0x7f3b000d5150 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.365+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3819481297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 msgr2=0x7f3b000d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.365+0000 7f3b07c7b640 1 --2- 172.21.15.12:0/3819481297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000d1a90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3b000b61c0 tx=0x7f3af802b770 comp rx=0 tx=0).stop 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.365+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3819481297 shutdown_connections 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.365+0000 7f3b07c7b640 1 --2- 172.21.15.12:0/3819481297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000d1a90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.365+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3819481297 >> 172.21.15.12:0/3819481297 conn(0x7f3b00067b40 msgr2=0x7f3b00067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.365+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3819481297 shutdown_connections 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.365+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3819481297 wait complete. 2024-04-15T15:21:37.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.366+0000 7f3b07c7b640 1 Processor -- start 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.366+0000 7f3b07c7b640 1 -- start start 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b07c7b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b000d15e0 0x7f3b000c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b07c7b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b07c7b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b0015e790 0x7f3b001f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b07c7b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3b000b7db0 con 0x7f3b000d5150 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b07c7b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3b000d45b0 con 0x7f3b000d15e0 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b07c7b640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3b000d29c0 con 0x7f3b0015e790 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46926/0 (socket says 172.21.15.12:46926) 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 -- 172.21.15.12:0/3706406147 learned_addr learned my addr 172.21.15.12:0/3706406147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05a17640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b000d15e0 0x7f3b000c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b06218640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b0015e790 0x7f3b001f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 -- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b0015e790 msgr2=0x7f3b001f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b0015e790 0x7f3b001f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 -- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b000d15e0 msgr2=0x7f3b000c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b000d15e0 0x7f3b000c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 -- 172.21.15.12:0/3706406147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b000c9ce0 con 0x7f3b000d5150 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05a17640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b000d15e0 0x7f3b000c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b05216640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000c9580 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3af000c460 tx=0x7f3af000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3af67fc640 1 -- 172.21.15.12:0/3706406147 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af0016020 con 0x7f3b000d5150 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3af67fc640 1 -- 172.21.15.12:0/3706406147 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af0012870 con 0x7f3b000d5150 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b001f55b0 con 0x7f3b000d5150 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3af67fc640 1 -- 172.21.15.12:0/3706406147 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af0012c60 con 0x7f3b000d5150 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.367+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b001f46e0 con 0x7f3b000d5150 2024-04-15T15:21:37.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.369+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b000689e0 con 0x7f3b000d5150 2024-04-15T15:21:37.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.369+0000 7f3af67fc640 1 -- 172.21.15.12:0/3706406147 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3af0019020 con 0x7f3b000d5150 2024-04-15T15:21:37.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.369+0000 7f3af67fc640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ae406c8f0 0x7f3ae406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.369+0000 7f3af67fc640 1 -- 172.21.15.12:0/3706406147 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3af008bb20 con 0x7f3b000d5150 2024-04-15T15:21:37.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.370+0000 7f3b05a17640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ae406c8f0 0x7f3ae406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.370+0000 7f3b05a17640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ae406c8f0 0x7f3ae406edb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0012e0 tx=0x7f3afc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:37.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.374+0000 7f3af67fc640 1 -- 172.21.15.12:0/3706406147 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3af0020030 con 0x7f3b000d5150 2024-04-15T15:21:37.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.555+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3b000ba2f0 con 0x7f3ae406c8f0 2024-04-15T15:21:37.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.563+0000 7f3af67fc640 1 -- 172.21.15.12:0/3706406147 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f3b000ba2f0 con 0x7f3ae406c8f0 2024-04-15T15:21:37.570 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ae406c8f0 msgr2=0x7f3ae406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ae406c8f0 0x7f3ae406edb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0012e0 tx=0x7f3afc009040 comp rx=0 tx=0).stop 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 msgr2=0x7f3b000c9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000c9580 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3af000c460 tx=0x7f3af000c920 comp rx=0 tx=0).stop 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 shutdown_connections 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3b0015e790 0x7f3b001f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b000d5150 0x7f3b000c9580 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ae406c8f0 0x7f3ae406edb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 --2- 172.21.15.12:0/3706406147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b000d15e0 0x7f3b000c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 >> 172.21.15.12:0/3706406147 conn(0x7f3b00067b40 msgr2=0x7f3b00074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 shutdown_connections 2024-04-15T15:21:37.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.566+0000 7f3b07c7b640 1 -- 172.21.15.12:0/3706406147 wait complete. 2024-04-15T15:21:37.579 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-15T15:21:36.593785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236140,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738580,"statfs":{"total":773094113280,"available":772852305920,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003138"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731482+0000","last_change":"2024-04-15T15:21:10.650644+0000","last_active":"2024-04-15T15:21:16.731482+0000","last_peered":"2024-04-15T15:21:16.731482+0000","last_clean":"2024-04-15T15:21:16.731482+0000","last_became_active":"2024-04-15T15:21:10.650315+0000","last_became_peered":"2024-04-15T15:21:10.650315+0000","last_unstale":"2024-04-15T15:21:16.731482+0000","last_undegraded":"2024-04-15T15:21:16.731482+0000","last_fullsized":"2024-04-15T15:21:16.731482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:27.659449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732827+0000","last_change":"2024-04-15T15:21:11.667138+0000","last_active":"2024-04-15T15:21:16.732827+0000","last_peered":"2024-04-15T15:21:16.732827+0000","last_clean":"2024-04-15T15:21:16.732827+0000","last_became_active":"2024-04-15T15:21:11.666799+0000","last_became_peered":"2024-04-15T15:21:11.666799+0000","last_unstale":"2024-04-15T15:21:16.732827+0000","last_undegraded":"2024-04-15T15:21:16.732827+0000","last_fullsized":"2024-04-15T15:21:16.732827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:24.468426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732274+0000","last_change":"2024-04-15T15:21:15.901408+0000","last_active":"2024-04-15T15:21:16.732274+0000","last_peered":"2024-04-15T15:21:16.732274+0000","last_clean":"2024-04-15T15:21:16.732274+0000","last_became_active":"2024-04-15T15:21:15.901032+0000","last_became_peered":"2024-04-15T15:21:15.901032+0000","last_unstale":"2024-04-15T15:21:16.732274+0000","last_undegraded":"2024-04-15T15:21:16.732274+0000","last_fullsized":"2024-04-15T15:21:16.732274+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:34.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733027+0000","last_change":"2024-04-15T15:21:10.636248+0000","last_active":"2024-04-15T15:21:16.733027+0000","last_peered":"2024-04-15T15:21:16.733027+0000","last_clean":"2024-04-15T15:21:16.733027+0000","last_became_active":"2024-04-15T15:21:10.635963+0000","last_became_peered":"2024-04-15T15:21:10.635963+0000","last_unstale":"2024-04-15T15:21:16.733027+0000","last_undegraded":"2024-04-15T15:21:16.733027+0000","last_fullsized":"2024-04-15T15:21:16.733027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:27.659449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731287+0000","last_change":"2024-04-15T15:21:11.665760+0000","last_active":"2024-04-15T15:21:16.731287+0000","last_peered":"2024-04-15T15:21:16.731287+0000","last_clean":"2024-04-15T15:21:16.731287+0000","last_became_active":"2024-04-15T15:21:11.665444+0000","last_became_peered":"2024-04-15T15:21:11.665444+0000","last_unstale":"2024-04-15T15:21:16.731287+0000","last_undegraded":"2024-04-15T15:21:16.731287+0000","last_fullsized":"2024-04-15T15:21:16.731287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.684953+0000","last_change":"2024-04-15T15:21:15.699988+0000","last_active":"2024-04-15T15:21:16.684953+0000","last_peered":"2024-04-15T15:21:16.684953+0000","last_clean":"2024-04-15T15:21:16.684953+0000","last_became_active":"2024-04-15T15:21:15.699333+0000","last_became_peered":"2024-04-15T15:21:15.699333+0000","last_unstale":"2024-04-15T15:21:16.684953+0000","last_undegraded":"2024-04-15T15:21:16.684953+0000","last_fullsized":"2024-04-15T15:21:16.684953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:26.928706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730928+0000","last_change":"2024-04-15T15:21:10.649466+0000","last_active":"2024-04-15T15:21:16.730928+0000","last_peered":"2024-04-15T15:21:16.730928+0000","last_clean":"2024-04-15T15:21:16.730928+0000","last_became_active":"2024-04-15T15:21:10.649182+0000","last_became_peered":"2024-04-15T15:21:10.649182+0000","last_unstale":"2024-04-15T15:21:16.730928+0000","last_undegraded":"2024-04-15T15:21:16.730928+0000","last_fullsized":"2024-04-15T15:21:16.730928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:01:16.245697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691301+0000","last_change":"2024-04-15T15:21:11.667011+0000","last_active":"2024-04-15T15:21:16.691301+0000","last_peered":"2024-04-15T15:21:16.691301+0000","last_clean":"2024-04-15T15:21:16.691301+0000","last_became_active":"2024-04-15T15:21:11.663992+0000","last_became_peered":"2024-04-15T15:21:11.663992+0000","last_unstale":"2024-04-15T15:21:16.691301+0000","last_undegraded":"2024-04-15T15:21:16.691301+0000","last_fullsized":"2024-04-15T15:21:16.691301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685332+0000","last_change":"2024-04-15T15:21:15.707372+0000","last_active":"2024-04-15T15:21:16.685332+0000","last_peered":"2024-04-15T15:21:16.685332+0000","last_clean":"2024-04-15T15:21:16.685332+0000","last_became_active":"2024-04-15T15:21:15.706978+0000","last_became_peered":"2024-04-15T15:21:15.706978+0000","last_unstale":"2024-04-15T15:21:16.685332+0000","last_undegraded":"2024-04-15T15:21:16.685332+0000","last_fullsized":"2024-04-15T15:21:16.685332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:32.327246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730471+0000","last_change":"2024-04-15T15:21:10.652340+0000","last_active":"2024-04-15T15:21:16.730471+0000","last_peered":"2024-04-15T15:21:16.730471+0000","last_clean":"2024-04-15T15:21:16.730471+0000","last_became_active":"2024-04-15T15:21:10.652016+0000","last_became_peered":"2024-04-15T15:21:10.652016+0000","last_unstale":"2024-04-15T15:21:16.730471+0000","last_undegraded":"2024-04-15T15:21:16.730471+0000","last_fullsized":"2024-04-15T15:21:16.730471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:54:08.208968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691646+0000","last_change":"2024-04-15T15:21:11.663781+0000","last_active":"2024-04-15T15:21:16.691646+0000","last_peered":"2024-04-15T15:21:16.691646+0000","last_clean":"2024-04-15T15:21:16.691646+0000","last_became_active":"2024-04-15T15:21:11.663232+0000","last_became_peered":"2024-04-15T15:21:11.663232+0000","last_unstale":"2024-04-15T15:21:16.691646+0000","last_undegraded":"2024-04-15T15:21:16.691646+0000","last_fullsized":"2024-04-15T15:21:16.691646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.568336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685080+0000","last_change":"2024-04-15T15:21:15.695846+0000","last_active":"2024-04-15T15:21:16.685080+0000","last_peered":"2024-04-15T15:21:16.685080+0000","last_clean":"2024-04-15T15:21:16.685080+0000","last_became_active":"2024-04-15T15:21:15.695426+0000","last_became_peered":"2024-04-15T15:21:15.695426+0000","last_unstale":"2024-04-15T15:21:16.685080+0000","last_undegraded":"2024-04-15T15:21:16.685080+0000","last_fullsized":"2024-04-15T15:21:16.685080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:28.518948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729851+0000","last_change":"2024-04-15T15:21:11.662485+0000","last_active":"2024-04-15T15:21:16.729851+0000","last_peered":"2024-04-15T15:21:16.729851+0000","last_clean":"2024-04-15T15:21:16.729851+0000","last_became_active":"2024-04-15T15:21:11.662183+0000","last_became_peered":"2024-04-15T15:21:11.662183+0000","last_unstale":"2024-04-15T15:21:16.729851+0000","last_undegraded":"2024-04-15T15:21:16.729851+0000","last_fullsized":"2024-04-15T15:21:16.729851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730547+0000","last_change":"2024-04-15T15:21:10.648295+0000","last_active":"2024-04-15T15:21:16.730547+0000","last_peered":"2024-04-15T15:21:16.730547+0000","last_clean":"2024-04-15T15:21:16.730547+0000","last_became_active":"2024-04-15T15:21:10.647985+0000","last_became_peered":"2024-04-15T15:21:10.647985+0000","last_unstale":"2024-04-15T15:21:16.730547+0000","last_undegraded":"2024-04-15T15:21:16.730547+0000","last_fullsized":"2024-04-15T15:21:16.730547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:52.730839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685664+0000","last_change":"2024-04-15T15:21:15.699611+0000","last_active":"2024-04-15T15:21:16.685664+0000","last_peered":"2024-04-15T15:21:16.685664+0000","last_clean":"2024-04-15T15:21:16.685664+0000","last_became_active":"2024-04-15T15:21:15.699129+0000","last_became_peered":"2024-04-15T15:21:15.699129+0000","last_unstale":"2024-04-15T15:21:16.685664+0000","last_undegraded":"2024-04-15T15:21:16.685664+0000","last_fullsized":"2024-04-15T15:21:16.685664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732647+0000","last_change":"2024-04-15T15:21:11.644620+0000","last_active":"2024-04-15T15:21:16.732647+0000","last_peered":"2024-04-15T15:21:16.732647+0000","last_clean":"2024-04-15T15:21:16.732647+0000","last_became_active":"2024-04-15T15:21:11.644273+0000","last_became_peered":"2024-04-15T15:21:11.644273+0000","last_unstale":"2024-04-15T15:21:16.732647+0000","last_undegraded":"2024-04-15T15:21:16.732647+0000","last_fullsized":"2024-04-15T15:21:16.732647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730443+0000","last_change":"2024-04-15T15:21:10.636457+0000","last_active":"2024-04-15T15:21:16.730443+0000","last_peered":"2024-04-15T15:21:16.730443+0000","last_clean":"2024-04-15T15:21:16.730443+0000","last_became_active":"2024-04-15T15:21:10.636143+0000","last_became_peered":"2024-04-15T15:21:10.636143+0000","last_unstale":"2024-04-15T15:21:16.730443+0000","last_undegraded":"2024-04-15T15:21:16.730443+0000","last_fullsized":"2024-04-15T15:21:16.730443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685684+0000","last_change":"2024-04-15T15:21:15.703570+0000","last_active":"2024-04-15T15:21:16.685684+0000","last_peered":"2024-04-15T15:21:16.685684+0000","last_clean":"2024-04-15T15:21:16.685684+0000","last_became_active":"2024-04-15T15:21:15.703066+0000","last_became_peered":"2024-04-15T15:21:15.703066+0000","last_unstale":"2024-04-15T15:21:16.685684+0000","last_undegraded":"2024-04-15T15:21:16.685684+0000","last_fullsized":"2024-04-15T15:21:16.685684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732393+0000","last_change":"2024-04-15T15:21:11.646745+0000","last_active":"2024-04-15T15:21:16.732393+0000","last_peered":"2024-04-15T15:21:16.732393+0000","last_clean":"2024-04-15T15:21:16.732393+0000","last_became_active":"2024-04-15T15:21:11.646375+0000","last_became_peered":"2024-04-15T15:21:11.646375+0000","last_unstale":"2024-04-15T15:21:16.732393+0000","last_undegraded":"2024-04-15T15:21:16.732393+0000","last_fullsized":"2024-04-15T15:21:16.732393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729475+0000","last_change":"2024-04-15T15:21:10.642462+0000","last_active":"2024-04-15T15:21:16.729475+0000","last_peered":"2024-04-15T15:21:16.729475+0000","last_clean":"2024-04-15T15:21:16.729475+0000","last_became_active":"2024-04-15T15:21:10.642168+0000","last_became_peered":"2024-04-15T15:21:10.642168+0000","last_unstale":"2024-04-15T15:21:16.729475+0000","last_undegraded":"2024-04-15T15:21:16.729475+0000","last_fullsized":"2024-04-15T15:21:16.729475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731042+0000","last_change":"2024-04-15T15:21:15.901600+0000","last_active":"2024-04-15T15:21:16.731042+0000","last_peered":"2024-04-15T15:21:16.731042+0000","last_clean":"2024-04-15T15:21:16.731042+0000","last_became_active":"2024-04-15T15:21:15.901233+0000","last_became_peered":"2024-04-15T15:21:15.901233+0000","last_unstale":"2024-04-15T15:21:16.731042+0000","last_undegraded":"2024-04-15T15:21:16.731042+0000","last_fullsized":"2024-04-15T15:21:16.731042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:30.138777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685208+0000","last_change":"2024-04-15T15:21:11.648506+0000","last_active":"2024-04-15T15:21:16.685208+0000","last_peered":"2024-04-15T15:21:16.685208+0000","last_clean":"2024-04-15T15:21:16.685208+0000","last_became_active":"2024-04-15T15:21:11.647972+0000","last_became_peered":"2024-04-15T15:21:11.647972+0000","last_unstale":"2024-04-15T15:21:16.685208+0000","last_undegraded":"2024-04-15T15:21:16.685208+0000","last_fullsized":"2024-04-15T15:21:16.685208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:09.853761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729694+0000","last_change":"2024-04-15T15:21:10.646955+0000","last_active":"2024-04-15T15:21:16.729694+0000","last_peered":"2024-04-15T15:21:16.729694+0000","last_clean":"2024-04-15T15:21:16.729694+0000","last_became_active":"2024-04-15T15:21:10.646308+0000","last_became_peered":"2024-04-15T15:21:10.646308+0000","last_unstale":"2024-04-15T15:21:16.729694+0000","last_undegraded":"2024-04-15T15:21:16.729694+0000","last_fullsized":"2024-04-15T15:21:16.729694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685568+0000","last_change":"2024-04-15T15:21:15.699814+0000","last_active":"2024-04-15T15:21:16.685568+0000","last_peered":"2024-04-15T15:21:16.685568+0000","last_clean":"2024-04-15T15:21:16.685568+0000","last_became_active":"2024-04-15T15:21:15.699341+0000","last_became_peered":"2024-04-15T15:21:15.699341+0000","last_unstale":"2024-04-15T15:21:16.685568+0000","last_undegraded":"2024-04-15T15:21:16.685568+0000","last_fullsized":"2024-04-15T15:21:16.685568+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:51.618858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731527+0000","last_change":"2024-04-15T15:21:11.646950+0000","last_active":"2024-04-15T15:21:16.731527+0000","last_peered":"2024-04-15T15:21:16.731527+0000","last_clean":"2024-04-15T15:21:16.731527+0000","last_became_active":"2024-04-15T15:21:11.646633+0000","last_became_peered":"2024-04-15T15:21:11.646633+0000","last_unstale":"2024-04-15T15:21:16.731527+0000","last_undegraded":"2024-04-15T15:21:16.731527+0000","last_fullsized":"2024-04-15T15:21:16.731527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:56.796018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730414+0000","last_change":"2024-04-15T15:21:10.646693+0000","last_active":"2024-04-15T15:21:16.730414+0000","last_peered":"2024-04-15T15:21:16.730414+0000","last_clean":"2024-04-15T15:21:16.730414+0000","last_became_active":"2024-04-15T15:21:10.646328+0000","last_became_peered":"2024-04-15T15:21:10.646328+0000","last_unstale":"2024-04-15T15:21:16.730414+0000","last_undegraded":"2024-04-15T15:21:16.730414+0000","last_fullsized":"2024-04-15T15:21:16.730414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685127+0000","last_change":"2024-04-15T15:21:15.902524+0000","last_active":"2024-04-15T15:21:16.685127+0000","last_peered":"2024-04-15T15:21:16.685127+0000","last_clean":"2024-04-15T15:21:16.685127+0000","last_became_active":"2024-04-15T15:21:15.901936+0000","last_became_peered":"2024-04-15T15:21:15.901936+0000","last_unstale":"2024-04-15T15:21:16.685127+0000","last_undegraded":"2024-04-15T15:21:16.685127+0000","last_fullsized":"2024-04-15T15:21:16.685127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:59.443259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733036+0000","last_change":"2024-04-15T15:21:11.657367+0000","last_active":"2024-04-15T15:21:16.733036+0000","last_peered":"2024-04-15T15:21:16.733036+0000","last_clean":"2024-04-15T15:21:16.733036+0000","last_became_active":"2024-04-15T15:21:11.657002+0000","last_became_peered":"2024-04-15T15:21:11.657002+0000","last_unstale":"2024-04-15T15:21:16.733036+0000","last_undegraded":"2024-04-15T15:21:16.733036+0000","last_fullsized":"2024-04-15T15:21:16.733036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:02.359534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731010+0000","last_change":"2024-04-15T15:21:10.644925+0000","last_active":"2024-04-15T15:21:16.731010+0000","last_peered":"2024-04-15T15:21:16.731010+0000","last_clean":"2024-04-15T15:21:16.731010+0000","last_became_active":"2024-04-15T15:21:10.644639+0000","last_became_peered":"2024-04-15T15:21:10.644639+0000","last_unstale":"2024-04-15T15:21:16.731010+0000","last_undegraded":"2024-04-15T15:21:16.731010+0000","last_fullsized":"2024-04-15T15:21:16.731010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732150+0000","last_change":"2024-04-15T15:21:15.902755+0000","last_active":"2024-04-15T15:21:16.732150+0000","last_peered":"2024-04-15T15:21:16.732150+0000","last_clean":"2024-04-15T15:21:16.732150+0000","last_became_active":"2024-04-15T15:21:15.902338+0000","last_became_peered":"2024-04-15T15:21:15.902338+0000","last_unstale":"2024-04-15T15:21:16.732150+0000","last_undegraded":"2024-04-15T15:21:16.732150+0000","last_fullsized":"2024-04-15T15:21:16.732150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:13.820849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730647+0000","last_change":"2024-04-15T15:21:11.654427+0000","last_active":"2024-04-15T15:21:16.730647+0000","last_peered":"2024-04-15T15:21:16.730647+0000","last_clean":"2024-04-15T15:21:16.730647+0000","last_became_active":"2024-04-15T15:21:11.654037+0000","last_became_peered":"2024-04-15T15:21:11.654037+0000","last_unstale":"2024-04-15T15:21:16.730647+0000","last_undegraded":"2024-04-15T15:21:16.730647+0000","last_fullsized":"2024-04-15T15:21:16.730647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731058+0000","last_change":"2024-04-15T15:21:10.649235+0000","last_active":"2024-04-15T15:21:16.731058+0000","last_peered":"2024-04-15T15:21:16.731058+0000","last_clean":"2024-04-15T15:21:16.731058+0000","last_became_active":"2024-04-15T15:21:10.648935+0000","last_became_peered":"2024-04-15T15:21:10.648935+0000","last_unstale":"2024-04-15T15:21:16.731058+0000","last_undegraded":"2024-04-15T15:21:16.731058+0000","last_fullsized":"2024-04-15T15:21:16.731058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:55.786608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729854+0000","last_change":"2024-04-15T15:21:15.899286+0000","last_active":"2024-04-15T15:21:16.729854+0000","last_peered":"2024-04-15T15:21:16.729854+0000","last_clean":"2024-04-15T15:21:16.729854+0000","last_became_active":"2024-04-15T15:21:15.898902+0000","last_became_peered":"2024-04-15T15:21:15.898902+0000","last_unstale":"2024-04-15T15:21:16.729854+0000","last_undegraded":"2024-04-15T15:21:16.729854+0000","last_fullsized":"2024-04-15T15:21:16.729854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691740+0000","last_change":"2024-04-15T15:21:11.661691+0000","last_active":"2024-04-15T15:21:16.691740+0000","last_peered":"2024-04-15T15:21:16.691740+0000","last_clean":"2024-04-15T15:21:16.691740+0000","last_became_active":"2024-04-15T15:21:11.661289+0000","last_became_peered":"2024-04-15T15:21:11.661289+0000","last_unstale":"2024-04-15T15:21:16.691740+0000","last_undegraded":"2024-04-15T15:21:16.691740+0000","last_fullsized":"2024-04-15T15:21:16.691740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:08.032185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730825+0000","last_change":"2024-04-15T15:21:10.637826+0000","last_active":"2024-04-15T15:21:16.730825+0000","last_peered":"2024-04-15T15:21:16.730825+0000","last_clean":"2024-04-15T15:21:16.730825+0000","last_became_active":"2024-04-15T15:21:10.637566+0000","last_became_peered":"2024-04-15T15:21:10.637566+0000","last_unstale":"2024-04-15T15:21:16.730825+0000","last_undegraded":"2024-04-15T15:21:16.730825+0000","last_fullsized":"2024-04-15T15:21:16.730825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:31.743027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729641+0000","last_change":"2024-04-15T15:21:15.707987+0000","last_active":"2024-04-15T15:21:16.729641+0000","last_peered":"2024-04-15T15:21:16.729641+0000","last_clean":"2024-04-15T15:21:16.729641+0000","last_became_active":"2024-04-15T15:21:15.707472+0000","last_became_peered":"2024-04-15T15:21:15.707472+0000","last_unstale":"2024-04-15T15:21:16.729641+0000","last_undegraded":"2024-04-15T15:21:16.729641+0000","last_fullsized":"2024-04-15T15:21:16.729641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685621+0000","last_change":"2024-04-15T15:21:11.644409+0000","last_active":"2024-04-15T15:21:16.685621+0000","last_peered":"2024-04-15T15:21:16.685621+0000","last_clean":"2024-04-15T15:21:16.685621+0000","last_became_active":"2024-04-15T15:21:11.644095+0000","last_became_peered":"2024-04-15T15:21:11.644095+0000","last_unstale":"2024-04-15T15:21:16.685621+0000","last_undegraded":"2024-04-15T15:21:16.685621+0000","last_fullsized":"2024-04-15T15:21:16.685621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:32.752437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691377+0000","last_change":"2024-04-15T15:21:10.646053+0000","last_active":"2024-04-15T15:21:16.691377+0000","last_peered":"2024-04-15T15:21:16.691377+0000","last_clean":"2024-04-15T15:21:16.691377+0000","last_became_active":"2024-04-15T15:21:10.645737+0000","last_became_peered":"2024-04-15T15:21:10.645737+0000","last_unstale":"2024-04-15T15:21:16.691377+0000","last_undegraded":"2024-04-15T15:21:16.691377+0000","last_fullsized":"2024-04-15T15:21:16.691377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730648+0000","last_change":"2024-04-15T15:21:15.699348+0000","last_active":"2024-04-15T15:21:16.730648+0000","last_peered":"2024-04-15T15:21:16.730648+0000","last_clean":"2024-04-15T15:21:16.730648+0000","last_became_active":"2024-04-15T15:21:15.698962+0000","last_became_peered":"2024-04-15T15:21:15.698962+0000","last_unstale":"2024-04-15T15:21:16.730648+0000","last_undegraded":"2024-04-15T15:21:16.730648+0000","last_fullsized":"2024-04-15T15:21:16.730648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:35.384335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729484+0000","last_change":"2024-04-15T15:21:11.653613+0000","last_active":"2024-04-15T15:21:16.729484+0000","last_peered":"2024-04-15T15:21:16.729484+0000","last_clean":"2024-04-15T15:21:16.729484+0000","last_became_active":"2024-04-15T15:21:11.653306+0000","last_became_peered":"2024-04-15T15:21:11.653306+0000","last_unstale":"2024-04-15T15:21:16.729484+0000","last_undegraded":"2024-04-15T15:21:16.729484+0000","last_fullsized":"2024-04-15T15:21:16.729484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731231+0000","last_change":"2024-04-15T15:21:10.636526+0000","last_active":"2024-04-15T15:21:16.731231+0000","last_peered":"2024-04-15T15:21:16.731231+0000","last_clean":"2024-04-15T15:21:16.731231+0000","last_became_active":"2024-04-15T15:21:10.636211+0000","last_became_peered":"2024-04-15T15:21:10.636211+0000","last_unstale":"2024-04-15T15:21:16.731231+0000","last_undegraded":"2024-04-15T15:21:16.731231+0000","last_fullsized":"2024-04-15T15:21:16.731231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731153+0000","last_change":"2024-04-15T15:21:15.706536+0000","last_active":"2024-04-15T15:21:16.731153+0000","last_peered":"2024-04-15T15:21:16.731153+0000","last_clean":"2024-04-15T15:21:16.731153+0000","last_became_active":"2024-04-15T15:21:15.706130+0000","last_became_peered":"2024-04-15T15:21:15.706130+0000","last_unstale":"2024-04-15T15:21:16.731153+0000","last_undegraded":"2024-04-15T15:21:16.731153+0000","last_fullsized":"2024-04-15T15:21:16.731153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:35.371338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729702+0000","last_change":"2024-04-15T15:21:11.649636+0000","last_active":"2024-04-15T15:21:16.729702+0000","last_peered":"2024-04-15T15:21:16.729702+0000","last_clean":"2024-04-15T15:21:16.729702+0000","last_became_active":"2024-04-15T15:21:11.649334+0000","last_became_peered":"2024-04-15T15:21:11.649334+0000","last_unstale":"2024-04-15T15:21:16.729702+0000","last_undegraded":"2024-04-15T15:21:16.729702+0000","last_fullsized":"2024-04-15T15:21:16.729702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:01:17.255107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730694+0000","last_change":"2024-04-15T15:21:10.647209+0000","last_active":"2024-04-15T15:21:16.730694+0000","last_peered":"2024-04-15T15:21:16.730694+0000","last_clean":"2024-04-15T15:21:16.730694+0000","last_became_active":"2024-04-15T15:21:10.646650+0000","last_became_peered":"2024-04-15T15:21:10.646650+0000","last_unstale":"2024-04-15T15:21:16.730694+0000","last_undegraded":"2024-04-15T15:21:16.730694+0000","last_fullsized":"2024-04-15T15:21:16.730694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:31.743027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731537+0000","last_change":"2024-04-15T15:21:15.901283+0000","last_active":"2024-04-15T15:21:16.731537+0000","last_peered":"2024-04-15T15:21:16.731537+0000","last_clean":"2024-04-15T15:21:16.731537+0000","last_became_active":"2024-04-15T15:21:15.900931+0000","last_became_peered":"2024-04-15T15:21:15.900931+0000","last_unstale":"2024-04-15T15:21:16.731537+0000","last_undegraded":"2024-04-15T15:21:16.731537+0000","last_fullsized":"2024-04-15T15:21:16.731537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:35.371338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732331+0000","last_change":"2024-04-15T15:21:11.664990+0000","last_active":"2024-04-15T15:21:16.732331+0000","last_peered":"2024-04-15T15:21:16.732331+0000","last_clean":"2024-04-15T15:21:16.732331+0000","last_became_active":"2024-04-15T15:21:11.664706+0000","last_became_peered":"2024-04-15T15:21:11.664706+0000","last_unstale":"2024-04-15T15:21:16.732331+0000","last_undegraded":"2024-04-15T15:21:16.732331+0000","last_fullsized":"2024-04-15T15:21:16.732331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691196+0000","last_change":"2024-04-15T15:21:10.645516+0000","last_active":"2024-04-15T15:21:16.691196+0000","last_peered":"2024-04-15T15:21:16.691196+0000","last_clean":"2024-04-15T15:21:16.691196+0000","last_became_active":"2024-04-15T15:21:10.645191+0000","last_became_peered":"2024-04-15T15:21:10.645191+0000","last_unstale":"2024-04-15T15:21:16.691196+0000","last_undegraded":"2024-04-15T15:21:16.691196+0000","last_fullsized":"2024-04-15T15:21:16.691196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730801+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:21:16.730801+0000","last_peered":"2024-04-15T15:21:16.730801+0000","last_clean":"2024-04-15T15:21:16.730801+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:21:16.730801+0000","last_undegraded":"2024-04-15T15:21:16.730801+0000","last_fullsized":"2024-04-15T15:21:16.730801+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730072+0000","last_change":"2024-04-15T15:21:15.901627+0000","last_active":"2024-04-15T15:21:16.730072+0000","last_peered":"2024-04-15T15:21:16.730072+0000","last_clean":"2024-04-15T15:21:16.730072+0000","last_became_active":"2024-04-15T15:21:15.901132+0000","last_became_peered":"2024-04-15T15:21:15.901132+0000","last_unstale":"2024-04-15T15:21:16.730072+0000","last_undegraded":"2024-04-15T15:21:16.730072+0000","last_fullsized":"2024-04-15T15:21:16.730072+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730781+0000","last_change":"2024-04-15T15:21:11.665709+0000","last_active":"2024-04-15T15:21:16.730781+0000","last_peered":"2024-04-15T15:21:16.730781+0000","last_clean":"2024-04-15T15:21:16.730781+0000","last_became_active":"2024-04-15T15:21:11.665254+0000","last_became_peered":"2024-04-15T15:21:11.665254+0000","last_unstale":"2024-04-15T15:21:16.730781+0000","last_undegraded":"2024-04-15T15:21:16.730781+0000","last_fullsized":"2024-04-15T15:21:16.730781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:31.333813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730984+0000","last_change":"2024-04-15T15:21:10.644131+0000","last_active":"2024-04-15T15:21:16.730984+0000","last_peered":"2024-04-15T15:21:16.730984+0000","last_clean":"2024-04-15T15:21:16.730984+0000","last_became_active":"2024-04-15T15:21:10.643844+0000","last_became_peered":"2024-04-15T15:21:10.643844+0000","last_unstale":"2024-04-15T15:21:16.730984+0000","last_undegraded":"2024-04-15T15:21:16.730984+0000","last_fullsized":"2024-04-15T15:21:16.730984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:16:25.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729900+0000","last_change":"2024-04-15T15:21:15.899284+0000","last_active":"2024-04-15T15:21:16.729900+0000","last_peered":"2024-04-15T15:21:16.729900+0000","last_clean":"2024-04-15T15:21:16.729900+0000","last_became_active":"2024-04-15T15:21:15.898822+0000","last_became_peered":"2024-04-15T15:21:15.898822+0000","last_unstale":"2024-04-15T15:21:16.729900+0000","last_undegraded":"2024-04-15T15:21:16.729900+0000","last_fullsized":"2024-04-15T15:21:16.729900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:24.117458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691307+0000","last_change":"2024-04-15T15:21:11.663221+0000","last_active":"2024-04-15T15:21:16.691307+0000","last_peered":"2024-04-15T15:21:16.691307+0000","last_clean":"2024-04-15T15:21:16.691307+0000","last_became_active":"2024-04-15T15:21:11.662827+0000","last_became_peered":"2024-04-15T15:21:11.662827+0000","last_unstale":"2024-04-15T15:21:16.691307+0000","last_undegraded":"2024-04-15T15:21:16.691307+0000","last_fullsized":"2024-04-15T15:21:16.691307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:15.476006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730374+0000","last_change":"2024-04-15T15:21:10.636999+0000","last_active":"2024-04-15T15:21:16.730374+0000","last_peered":"2024-04-15T15:21:16.730374+0000","last_clean":"2024-04-15T15:21:16.730374+0000","last_became_active":"2024-04-15T15:21:10.636729+0000","last_became_peered":"2024-04-15T15:21:10.636729+0000","last_unstale":"2024-04-15T15:21:16.730374+0000","last_undegraded":"2024-04-15T15:21:16.730374+0000","last_fullsized":"2024-04-15T15:21:16.730374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730116+0000","last_change":"2024-04-15T15:21:15.708002+0000","last_active":"2024-04-15T15:21:16.730116+0000","last_peered":"2024-04-15T15:21:16.730116+0000","last_clean":"2024-04-15T15:21:16.730116+0000","last_became_active":"2024-04-15T15:21:15.707636+0000","last_became_peered":"2024-04-15T15:21:15.707636+0000","last_unstale":"2024-04-15T15:21:16.730116+0000","last_undegraded":"2024-04-15T15:21:16.730116+0000","last_fullsized":"2024-04-15T15:21:16.730116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730705+0000","last_change":"2024-04-15T15:21:11.662157+0000","last_active":"2024-04-15T15:21:16.730705+0000","last_peered":"2024-04-15T15:21:16.730705+0000","last_clean":"2024-04-15T15:21:16.730705+0000","last_became_active":"2024-04-15T15:21:11.661885+0000","last_became_peered":"2024-04-15T15:21:11.661885+0000","last_unstale":"2024-04-15T15:21:16.730705+0000","last_undegraded":"2024-04-15T15:21:16.730705+0000","last_fullsized":"2024-04-15T15:21:16.730705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:30.561749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731734+0000","last_change":"2024-04-15T15:21:10.635657+0000","last_active":"2024-04-15T15:21:16.731734+0000","last_peered":"2024-04-15T15:21:16.731734+0000","last_clean":"2024-04-15T15:21:16.731734+0000","last_became_active":"2024-04-15T15:21:10.635339+0000","last_became_peered":"2024-04-15T15:21:10.635339+0000","last_unstale":"2024-04-15T15:21:16.731734+0000","last_undegraded":"2024-04-15T15:21:16.731734+0000","last_fullsized":"2024-04-15T15:21:16.731734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690911+0000","last_change":"2024-04-15T15:21:15.902642+0000","last_active":"2024-04-15T15:21:16.690911+0000","last_peered":"2024-04-15T15:21:16.690911+0000","last_clean":"2024-04-15T15:21:16.690911+0000","last_became_active":"2024-04-15T15:21:15.902252+0000","last_became_peered":"2024-04-15T15:21:15.902252+0000","last_unstale":"2024-04-15T15:21:16.690911+0000","last_undegraded":"2024-04-15T15:21:16.690911+0000","last_fullsized":"2024-04-15T15:21:16.690911+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:13.820849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730900+0000","last_change":"2024-04-15T15:21:11.658612+0000","last_active":"2024-04-15T15:21:16.730900+0000","last_peered":"2024-04-15T15:21:16.730900+0000","last_clean":"2024-04-15T15:21:16.730900+0000","last_became_active":"2024-04-15T15:21:11.658312+0000","last_became_peered":"2024-04-15T15:21:11.658312+0000","last_unstale":"2024-04-15T15:21:16.730900+0000","last_undegraded":"2024-04-15T15:21:16.730900+0000","last_fullsized":"2024-04-15T15:21:16.730900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:15.476006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685731+0000","last_change":"2024-04-15T15:21:10.636964+0000","last_active":"2024-04-15T15:21:16.685731+0000","last_peered":"2024-04-15T15:21:16.685731+0000","last_clean":"2024-04-15T15:21:16.685731+0000","last_became_active":"2024-04-15T15:21:10.636641+0000","last_became_peered":"2024-04-15T15:21:10.636641+0000","last_unstale":"2024-04-15T15:21:16.685731+0000","last_undegraded":"2024-04-15T15:21:16.685731+0000","last_fullsized":"2024-04-15T15:21:16.685731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:27:05.012468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.684974+0000","last_change":"2024-04-15T15:21:15.706953+0000","last_active":"2024-04-15T15:21:16.684974+0000","last_peered":"2024-04-15T15:21:16.684974+0000","last_clean":"2024-04-15T15:21:16.684974+0000","last_became_active":"2024-04-15T15:21:15.706465+0000","last_became_peered":"2024-04-15T15:21:15.706465+0000","last_unstale":"2024-04-15T15:21:16.684974+0000","last_undegraded":"2024-04-15T15:21:16.684974+0000","last_fullsized":"2024-04-15T15:21:16.684974+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:06.410056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731056+0000","last_change":"2024-04-15T15:21:11.658373+0000","last_active":"2024-04-15T15:21:16.731056+0000","last_peered":"2024-04-15T15:21:16.731056+0000","last_clean":"2024-04-15T15:21:16.731056+0000","last_became_active":"2024-04-15T15:21:11.658069+0000","last_became_peered":"2024-04-15T15:21:11.658069+0000","last_unstale":"2024-04-15T15:21:16.731056+0000","last_undegraded":"2024-04-15T15:21:16.731056+0000","last_fullsized":"2024-04-15T15:21:16.731056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:31.320816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730149+0000","last_change":"2024-04-15T15:21:10.637607+0000","last_active":"2024-04-15T15:21:16.730149+0000","last_peered":"2024-04-15T15:21:16.730149+0000","last_clean":"2024-04-15T15:21:16.730149+0000","last_became_active":"2024-04-15T15:21:10.637277+0000","last_became_peered":"2024-04-15T15:21:10.637277+0000","last_unstale":"2024-04-15T15:21:16.730149+0000","last_undegraded":"2024-04-15T15:21:16.730149+0000","last_fullsized":"2024-04-15T15:21:16.730149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:03:48.192888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731366+0000","last_change":"2024-04-15T15:21:15.696964+0000","last_active":"2024-04-15T15:21:16.731366+0000","last_peered":"2024-04-15T15:21:16.731366+0000","last_clean":"2024-04-15T15:21:16.731366+0000","last_became_active":"2024-04-15T15:21:15.696559+0000","last_became_peered":"2024-04-15T15:21:15.696559+0000","last_unstale":"2024-04-15T15:21:16.731366+0000","last_undegraded":"2024-04-15T15:21:16.731366+0000","last_fullsized":"2024-04-15T15:21:16.731366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:35.384335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729725+0000","last_change":"2024-04-15T15:21:11.661036+0000","last_active":"2024-04-15T15:21:16.729725+0000","last_peered":"2024-04-15T15:21:16.729725+0000","last_clean":"2024-04-15T15:21:16.729725+0000","last_became_active":"2024-04-15T15:21:11.660616+0000","last_became_peered":"2024-04-15T15:21:11.660616+0000","last_unstale":"2024-04-15T15:21:16.729725+0000","last_undegraded":"2024-04-15T15:21:16.729725+0000","last_fullsized":"2024-04-15T15:21:16.729725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729989+0000","last_change":"2024-04-15T15:21:10.644599+0000","last_active":"2024-04-15T15:21:16.729989+0000","last_peered":"2024-04-15T15:21:16.729989+0000","last_clean":"2024-04-15T15:21:16.729989+0000","last_became_active":"2024-04-15T15:21:10.644274+0000","last_became_peered":"2024-04-15T15:21:10.644274+0000","last_unstale":"2024-04-15T15:21:16.729989+0000","last_undegraded":"2024-04-15T15:21:16.729989+0000","last_fullsized":"2024-04-15T15:21:16.729989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732952+0000","last_change":"2024-04-15T15:21:15.901481+0000","last_active":"2024-04-15T15:21:16.732952+0000","last_peered":"2024-04-15T15:21:16.732952+0000","last_clean":"2024-04-15T15:21:16.732952+0000","last_became_active":"2024-04-15T15:21:15.901096+0000","last_became_peered":"2024-04-15T15:21:15.901096+0000","last_unstale":"2024-04-15T15:21:16.732952+0000","last_undegraded":"2024-04-15T15:21:16.732952+0000","last_fullsized":"2024-04-15T15:21:16.732952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:38.490295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730312+0000","last_change":"2024-04-15T15:21:11.658341+0000","last_active":"2024-04-15T15:21:16.730312+0000","last_peered":"2024-04-15T15:21:16.730312+0000","last_clean":"2024-04-15T15:21:16.730312+0000","last_became_active":"2024-04-15T15:21:11.658024+0000","last_became_peered":"2024-04-15T15:21:11.658024+0000","last_unstale":"2024-04-15T15:21:16.730312+0000","last_undegraded":"2024-04-15T15:21:16.730312+0000","last_fullsized":"2024-04-15T15:21:16.730312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.568336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691308+0000","last_change":"2024-04-15T15:21:10.645818+0000","last_active":"2024-04-15T15:21:16.691308+0000","last_peered":"2024-04-15T15:21:16.691308+0000","last_clean":"2024-04-15T15:21:16.691308+0000","last_became_active":"2024-04-15T15:21:10.645371+0000","last_became_peered":"2024-04-15T15:21:10.645371+0000","last_unstale":"2024-04-15T15:21:16.691308+0000","last_undegraded":"2024-04-15T15:21:16.691308+0000","last_fullsized":"2024-04-15T15:21:16.691308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733065+0000","last_change":"2024-04-15T15:21:15.899911+0000","last_active":"2024-04-15T15:21:16.733065+0000","last_peered":"2024-04-15T15:21:16.733065+0000","last_clean":"2024-04-15T15:21:16.733065+0000","last_became_active":"2024-04-15T15:21:15.899428+0000","last_became_peered":"2024-04-15T15:21:15.899428+0000","last_unstale":"2024-04-15T15:21:16.733065+0000","last_undegraded":"2024-04-15T15:21:16.733065+0000","last_fullsized":"2024-04-15T15:21:16.733065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:18:40.268133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731100+0000","last_change":"2024-04-15T15:21:11.659032+0000","last_active":"2024-04-15T15:21:16.731100+0000","last_peered":"2024-04-15T15:21:16.731100+0000","last_clean":"2024-04-15T15:21:16.731100+0000","last_became_active":"2024-04-15T15:21:11.658769+0000","last_became_peered":"2024-04-15T15:21:11.658769+0000","last_unstale":"2024-04-15T15:21:16.731100+0000","last_undegraded":"2024-04-15T15:21:16.731100+0000","last_fullsized":"2024-04-15T15:21:16.731100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730358+0000","last_change":"2024-04-15T15:21:10.640715+0000","last_active":"2024-04-15T15:21:16.730358+0000","last_peered":"2024-04-15T15:21:16.730358+0000","last_clean":"2024-04-15T15:21:16.730358+0000","last_became_active":"2024-04-15T15:21:10.640364+0000","last_became_peered":"2024-04-15T15:21:10.640364+0000","last_unstale":"2024-04-15T15:21:16.730358+0000","last_undegraded":"2024-04-15T15:21:16.730358+0000","last_fullsized":"2024-04-15T15:21:16.730358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:54:53.368896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691556+0000","last_change":"2024-04-15T15:21:15.901574+0000","last_active":"2024-04-15T15:21:16.691556+0000","last_peered":"2024-04-15T15:21:16.691556+0000","last_clean":"2024-04-15T15:21:16.691556+0000","last_became_active":"2024-04-15T15:21:15.901144+0000","last_became_peered":"2024-04-15T15:21:15.901144+0000","last_unstale":"2024-04-15T15:21:16.691556+0000","last_undegraded":"2024-04-15T15:21:16.691556+0000","last_fullsized":"2024-04-15T15:21:16.691556+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:34.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730711+0000","last_change":"2024-04-15T15:21:11.655252+0000","last_active":"2024-04-15T15:21:16.730711+0000","last_peered":"2024-04-15T15:21:16.730711+0000","last_clean":"2024-04-15T15:21:16.730711+0000","last_became_active":"2024-04-15T15:21:11.654839+0000","last_became_peered":"2024-04-15T15:21:11.654839+0000","last_unstale":"2024-04-15T15:21:16.730711+0000","last_undegraded":"2024-04-15T15:21:16.730711+0000","last_fullsized":"2024-04-15T15:21:16.730711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:08.032185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:21.251862+0000","last_change":"2024-04-15T15:21:10.645952+0000","last_active":"2024-04-15T15:21:21.251862+0000","last_peered":"2024-04-15T15:21:21.251862+0000","last_clean":"2024-04-15T15:21:21.251862+0000","last_became_active":"2024-04-15T15:21:10.645618+0000","last_became_peered":"2024-04-15T15:21:10.645618+0000","last_unstale":"2024-04-15T15:21:21.251862+0000","last_undegraded":"2024-04-15T15:21:21.251862+0000","last_fullsized":"2024-04-15T15:21:21.251862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730096+0000","last_change":"2024-04-15T15:21:15.703360+0000","last_active":"2024-04-15T15:21:16.730096+0000","last_peered":"2024-04-15T15:21:16.730096+0000","last_clean":"2024-04-15T15:21:16.730096+0000","last_became_active":"2024-04-15T15:21:15.702868+0000","last_became_peered":"2024-04-15T15:21:15.702868+0000","last_unstale":"2024-04-15T15:21:16.730096+0000","last_undegraded":"2024-04-15T15:21:16.730096+0000","last_fullsized":"2024-04-15T15:21:16.730096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685743+0000","last_change":"2024-04-15T15:21:11.659799+0000","last_active":"2024-04-15T15:21:16.685743+0000","last_peered":"2024-04-15T15:21:16.685743+0000","last_clean":"2024-04-15T15:21:16.685743+0000","last_became_active":"2024-04-15T15:21:11.659521+0000","last_became_peered":"2024-04-15T15:21:11.659521+0000","last_unstale":"2024-04-15T15:21:16.685743+0000","last_undegraded":"2024-04-15T15:21:16.685743+0000","last_fullsized":"2024-04-15T15:21:16.685743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:38.472181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730267+0000","last_change":"2024-04-15T15:21:10.645938+0000","last_active":"2024-04-15T15:21:16.730267+0000","last_peered":"2024-04-15T15:21:16.730267+0000","last_clean":"2024-04-15T15:21:16.730267+0000","last_became_active":"2024-04-15T15:21:10.645642+0000","last_became_peered":"2024-04-15T15:21:10.645642+0000","last_unstale":"2024-04-15T15:21:16.730267+0000","last_undegraded":"2024-04-15T15:21:16.730267+0000","last_fullsized":"2024-04-15T15:21:16.730267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685481+0000","last_change":"2024-04-15T15:21:15.705076+0000","last_active":"2024-04-15T15:21:16.685481+0000","last_peered":"2024-04-15T15:21:16.685481+0000","last_clean":"2024-04-15T15:21:16.685481+0000","last_became_active":"2024-04-15T15:21:15.704627+0000","last_became_peered":"2024-04-15T15:21:15.704627+0000","last_unstale":"2024-04-15T15:21:16.685481+0000","last_undegraded":"2024-04-15T15:21:16.685481+0000","last_fullsized":"2024-04-15T15:21:16.685481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:24.117458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731387+0000","last_change":"2024-04-15T15:21:11.665544+0000","last_active":"2024-04-15T15:21:16.731387+0000","last_peered":"2024-04-15T15:21:16.731387+0000","last_clean":"2024-04-15T15:21:16.731387+0000","last_became_active":"2024-04-15T15:21:11.665263+0000","last_became_peered":"2024-04-15T15:21:11.665263+0000","last_unstale":"2024-04-15T15:21:16.731387+0000","last_undegraded":"2024-04-15T15:21:16.731387+0000","last_fullsized":"2024-04-15T15:21:16.731387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691599+0000","last_change":"2024-04-15T15:21:10.648496+0000","last_active":"2024-04-15T15:21:16.691599+0000","last_peered":"2024-04-15T15:21:16.691599+0000","last_clean":"2024-04-15T15:21:16.691599+0000","last_became_active":"2024-04-15T15:21:10.648222+0000","last_became_peered":"2024-04-15T15:21:10.648222+0000","last_unstale":"2024-04-15T15:21:16.691599+0000","last_undegraded":"2024-04-15T15:21:16.691599+0000","last_fullsized":"2024-04-15T15:21:16.691599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732523+0000","last_change":"2024-04-15T15:21:15.699583+0000","last_active":"2024-04-15T15:21:16.732523+0000","last_peered":"2024-04-15T15:21:16.732523+0000","last_clean":"2024-04-15T15:21:16.732523+0000","last_became_active":"2024-04-15T15:21:15.699119+0000","last_became_peered":"2024-04-15T15:21:15.699119+0000","last_unstale":"2024-04-15T15:21:16.732523+0000","last_undegraded":"2024-04-15T15:21:16.732523+0000","last_fullsized":"2024-04-15T15:21:16.732523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:41:57.140691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691502+0000","last_change":"2024-04-15T15:21:11.663796+0000","last_active":"2024-04-15T15:21:16.691502+0000","last_peered":"2024-04-15T15:21:16.691502+0000","last_clean":"2024-04-15T15:21:16.691502+0000","last_became_active":"2024-04-15T15:21:11.663502+0000","last_became_peered":"2024-04-15T15:21:11.663502+0000","last_unstale":"2024-04-15T15:21:16.691502+0000","last_undegraded":"2024-04-15T15:21:16.691502+0000","last_fullsized":"2024-04-15T15:21:16.691502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:31.333813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730207+0000","last_change":"2024-04-15T15:21:10.651176+0000","last_active":"2024-04-15T15:21:16.730207+0000","last_peered":"2024-04-15T15:21:16.730207+0000","last_clean":"2024-04-15T15:21:16.730207+0000","last_became_active":"2024-04-15T15:21:10.650301+0000","last_became_peered":"2024-04-15T15:21:10.650301+0000","last_unstale":"2024-04-15T15:21:16.730207+0000","last_undegraded":"2024-04-15T15:21:16.730207+0000","last_fullsized":"2024-04-15T15:21:16.730207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732313+0000","last_change":"2024-04-15T15:21:15.899209+0000","last_active":"2024-04-15T15:21:16.732313+0000","last_peered":"2024-04-15T15:21:16.732313+0000","last_clean":"2024-04-15T15:21:16.732313+0000","last_became_active":"2024-04-15T15:21:15.898799+0000","last_became_peered":"2024-04-15T15:21:15.898799+0000","last_unstale":"2024-04-15T15:21:16.732313+0000","last_undegraded":"2024-04-15T15:21:16.732313+0000","last_fullsized":"2024-04-15T15:21:16.732313+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:16.350687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731404+0000","last_change":"2024-04-15T15:21:15.902104+0000","last_active":"2024-04-15T15:21:16.731404+0000","last_peered":"2024-04-15T15:21:16.731404+0000","last_clean":"2024-04-15T15:21:16.731404+0000","last_became_active":"2024-04-15T15:21:15.901711+0000","last_became_peered":"2024-04-15T15:21:15.901711+0000","last_unstale":"2024-04-15T15:21:16.731404+0000","last_undegraded":"2024-04-15T15:21:16.731404+0000","last_fullsized":"2024-04-15T15:21:16.731404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:30.138777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729279+0000","last_change":"2024-04-15T15:21:11.649924+0000","last_active":"2024-04-15T15:21:16.729279+0000","last_peered":"2024-04-15T15:21:16.729279+0000","last_clean":"2024-04-15T15:21:16.729279+0000","last_became_active":"2024-04-15T15:21:11.649654+0000","last_became_peered":"2024-04-15T15:21:11.649654+0000","last_unstale":"2024-04-15T15:21:16.729279+0000","last_undegraded":"2024-04-15T15:21:16.729279+0000","last_fullsized":"2024-04-15T15:21:16.729279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691073+0000","last_change":"2024-04-15T15:21:10.647906+0000","last_active":"2024-04-15T15:21:16.691073+0000","last_peered":"2024-04-15T15:21:16.691073+0000","last_clean":"2024-04-15T15:21:16.691073+0000","last_became_active":"2024-04-15T15:21:10.647594+0000","last_became_peered":"2024-04-15T15:21:10.647594+0000","last_unstale":"2024-04-15T15:21:16.691073+0000","last_undegraded":"2024-04-15T15:21:16.691073+0000","last_fullsized":"2024-04-15T15:21:16.691073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729962+0000","last_change":"2024-04-15T15:21:15.707054+0000","last_active":"2024-04-15T15:21:16.729962+0000","last_peered":"2024-04-15T15:21:16.729962+0000","last_clean":"2024-04-15T15:21:16.729962+0000","last_became_active":"2024-04-15T15:21:15.706636+0000","last_became_peered":"2024-04-15T15:21:15.706636+0000","last_unstale":"2024-04-15T15:21:16.729962+0000","last_undegraded":"2024-04-15T15:21:16.729962+0000","last_fullsized":"2024-04-15T15:21:16.729962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:51.618858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732275+0000","last_change":"2024-04-15T15:21:11.644995+0000","last_active":"2024-04-15T15:21:16.732275+0000","last_peered":"2024-04-15T15:21:16.732275+0000","last_clean":"2024-04-15T15:21:16.732275+0000","last_became_active":"2024-04-15T15:21:11.644654+0000","last_became_peered":"2024-04-15T15:21:11.644654+0000","last_unstale":"2024-04-15T15:21:16.732275+0000","last_undegraded":"2024-04-15T15:21:16.732275+0000","last_fullsized":"2024-04-15T15:21:16.732275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732479+0000","last_change":"2024-04-15T15:21:10.645892+0000","last_active":"2024-04-15T15:21:16.732479+0000","last_peered":"2024-04-15T15:21:16.732479+0000","last_clean":"2024-04-15T15:21:16.732479+0000","last_became_active":"2024-04-15T15:21:10.645615+0000","last_became_peered":"2024-04-15T15:21:10.645615+0000","last_unstale":"2024-04-15T15:21:16.732479+0000","last_undegraded":"2024-04-15T15:21:16.732479+0000","last_fullsized":"2024-04-15T15:21:16.732479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:16:25.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730156+0000","last_change":"2024-04-15T15:21:10.646124+0000","last_active":"2024-04-15T15:21:16.730156+0000","last_peered":"2024-04-15T15:21:16.730156+0000","last_clean":"2024-04-15T15:21:16.730156+0000","last_became_active":"2024-04-15T15:21:10.644818+0000","last_became_peered":"2024-04-15T15:21:10.644818+0000","last_unstale":"2024-04-15T15:21:16.730156+0000","last_undegraded":"2024-04-15T15:21:16.730156+0000","last_fullsized":"2024-04-15T15:21:16.730156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730262+0000","last_change":"2024-04-15T15:21:11.648824+0000","last_active":"2024-04-15T15:21:16.730262+0000","last_peered":"2024-04-15T15:21:16.730262+0000","last_clean":"2024-04-15T15:21:16.730262+0000","last_became_active":"2024-04-15T15:21:11.648490+0000","last_became_peered":"2024-04-15T15:21:11.648490+0000","last_unstale":"2024-04-15T15:21:16.730262+0000","last_undegraded":"2024-04-15T15:21:16.730262+0000","last_fullsized":"2024-04-15T15:21:16.730262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730777+0000","last_change":"2024-04-15T15:21:15.699586+0000","last_active":"2024-04-15T15:21:16.730777+0000","last_peered":"2024-04-15T15:21:16.730777+0000","last_clean":"2024-04-15T15:21:16.730777+0000","last_became_active":"2024-04-15T15:21:15.699130+0000","last_became_peered":"2024-04-15T15:21:15.699130+0000","last_unstale":"2024-04-15T15:21:16.730777+0000","last_undegraded":"2024-04-15T15:21:16.730777+0000","last_fullsized":"2024-04-15T15:21:16.730777+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:47.552269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732768+0000","last_change":"2024-04-15T15:21:10.641801+0000","last_active":"2024-04-15T15:21:16.732768+0000","last_peered":"2024-04-15T15:21:16.732768+0000","last_clean":"2024-04-15T15:21:16.732768+0000","last_became_active":"2024-04-15T15:21:10.641511+0000","last_became_peered":"2024-04-15T15:21:10.641511+0000","last_unstale":"2024-04-15T15:21:16.732768+0000","last_undegraded":"2024-04-15T15:21:16.732768+0000","last_fullsized":"2024-04-15T15:21:16.732768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:03:48.192888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685152+0000","last_change":"2024-04-15T15:21:11.650188+0000","last_active":"2024-04-15T15:21:16.685152+0000","last_peered":"2024-04-15T15:21:16.685152+0000","last_clean":"2024-04-15T15:21:16.685152+0000","last_became_active":"2024-04-15T15:21:11.649838+0000","last_became_peered":"2024-04-15T15:21:11.649838+0000","last_unstale":"2024-04-15T15:21:16.685152+0000","last_undegraded":"2024-04-15T15:21:16.685152+0000","last_fullsized":"2024-04-15T15:21:16.685152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:12.989671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732888+0000","last_change":"2024-04-15T15:21:15.699639+0000","last_active":"2024-04-15T15:21:16.732888+0000","last_peered":"2024-04-15T15:21:16.732888+0000","last_clean":"2024-04-15T15:21:16.732888+0000","last_became_active":"2024-04-15T15:21:15.699252+0000","last_became_peered":"2024-04-15T15:21:15.699252+0000","last_unstale":"2024-04-15T15:21:16.732888+0000","last_undegraded":"2024-04-15T15:21:16.732888+0000","last_fullsized":"2024-04-15T15:21:16.732888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:02.471663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729618+0000","last_change":"2024-04-15T15:21:11.662366+0000","last_active":"2024-04-15T15:21:16.729618+0000","last_peered":"2024-04-15T15:21:16.729618+0000","last_clean":"2024-04-15T15:21:16.729618+0000","last_became_active":"2024-04-15T15:21:11.662082+0000","last_became_peered":"2024-04-15T15:21:11.662082+0000","last_unstale":"2024-04-15T15:21:16.729618+0000","last_undegraded":"2024-04-15T15:21:16.729618+0000","last_fullsized":"2024-04-15T15:21:16.729618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731145+0000","last_change":"2024-04-15T15:21:11.667867+0000","last_active":"2024-04-15T15:21:16.731145+0000","last_peered":"2024-04-15T15:21:16.731145+0000","last_clean":"2024-04-15T15:21:16.731145+0000","last_became_active":"2024-04-15T15:21:11.667664+0000","last_became_peered":"2024-04-15T15:21:11.667664+0000","last_unstale":"2024-04-15T15:21:16.731145+0000","last_undegraded":"2024-04-15T15:21:16.731145+0000","last_fullsized":"2024-04-15T15:21:16.731145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:32.752437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691237+0000","last_change":"2024-04-15T15:21:11.662949+0000","last_active":"2024-04-15T15:21:16.691237+0000","last_peered":"2024-04-15T15:21:16.691237+0000","last_clean":"2024-04-15T15:21:16.691237+0000","last_became_active":"2024-04-15T15:21:11.662014+0000","last_became_peered":"2024-04-15T15:21:11.662014+0000","last_unstale":"2024-04-15T15:21:16.691237+0000","last_undegraded":"2024-04-15T15:21:16.691237+0000","last_fullsized":"2024-04-15T15:21:16.691237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:20.066936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730170+0000","last_change":"2024-04-15T15:21:11.643845+0000","last_active":"2024-04-15T15:21:16.730170+0000","last_peered":"2024-04-15T15:21:16.730170+0000","last_clean":"2024-04-15T15:21:16.730170+0000","last_became_active":"2024-04-15T15:21:11.643459+0000","last_became_peered":"2024-04-15T15:21:11.643459+0000","last_unstale":"2024-04-15T15:21:16.730170+0000","last_undegraded":"2024-04-15T15:21:16.730170+0000","last_fullsized":"2024-04-15T15:21:16.730170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:43.501747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733169+0000","last_change":"2024-04-15T15:21:11.649864+0000","last_active":"2024-04-15T15:21:16.733169+0000","last_peered":"2024-04-15T15:21:16.733169+0000","last_clean":"2024-04-15T15:21:16.733169+0000","last_became_active":"2024-04-15T15:21:11.649174+0000","last_became_peered":"2024-04-15T15:21:11.649174+0000","last_unstale":"2024-04-15T15:21:16.733169+0000","last_undegraded":"2024-04-15T15:21:16.733169+0000","last_fullsized":"2024-04-15T15:21:16.733169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:55.392737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730146+0000","last_change":"2024-04-15T15:21:11.649917+0000","last_active":"2024-04-15T15:21:16.730146+0000","last_peered":"2024-04-15T15:21:16.730146+0000","last_clean":"2024-04-15T15:21:16.730146+0000","last_became_active":"2024-04-15T15:21:11.649627+0000","last_became_peered":"2024-04-15T15:21:11.649627+0000","last_unstale":"2024-04-15T15:21:16.730146+0000","last_undegraded":"2024-04-15T15:21:16.730146+0000","last_fullsized":"2024-04-15T15:21:16.730146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730543+0000","last_change":"2024-04-15T15:21:11.666666+0000","last_active":"2024-04-15T15:21:16.730543+0000","last_peered":"2024-04-15T15:21:16.730543+0000","last_clean":"2024-04-15T15:21:16.730543+0000","last_became_active":"2024-04-15T15:21:11.666354+0000","last_became_peered":"2024-04-15T15:21:11.666354+0000","last_unstale":"2024-04-15T15:21:16.730543+0000","last_undegraded":"2024-04-15T15:21:16.730543+0000","last_fullsized":"2024-04-15T15:21:16.730543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732965+0000","last_change":"2024-04-15T15:21:11.649875+0000","last_active":"2024-04-15T15:21:16.732965+0000","last_peered":"2024-04-15T15:21:16.732965+0000","last_clean":"2024-04-15T15:21:16.732965+0000","last_became_active":"2024-04-15T15:21:11.649492+0000","last_became_peered":"2024-04-15T15:21:11.649492+0000","last_unstale":"2024-04-15T15:21:16.732965+0000","last_undegraded":"2024-04-15T15:21:16.732965+0000","last_fullsized":"2024-04-15T15:21:16.732965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690980+0000","last_change":"2024-04-15T15:21:11.663545+0000","last_active":"2024-04-15T15:21:16.690980+0000","last_peered":"2024-04-15T15:21:16.690980+0000","last_clean":"2024-04-15T15:21:16.690980+0000","last_became_active":"2024-04-15T15:21:11.662983+0000","last_became_peered":"2024-04-15T15:21:11.662983+0000","last_unstale":"2024-04-15T15:21:16.690980+0000","last_undegraded":"2024-04-15T15:21:16.690980+0000","last_fullsized":"2024-04-15T15:21:16.690980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:24.468426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732374+0000","last_change":"2024-04-15T15:21:11.649581+0000","last_active":"2024-04-15T15:21:16.732374+0000","last_peered":"2024-04-15T15:21:16.732374+0000","last_clean":"2024-04-15T15:21:16.732374+0000","last_became_active":"2024-04-15T15:21:11.649255+0000","last_became_peered":"2024-04-15T15:21:11.649255+0000","last_unstale":"2024-04-15T15:21:16.732374+0000","last_undegraded":"2024-04-15T15:21:16.732374+0000","last_fullsized":"2024-04-15T15:21:16.732374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691111+0000","last_change":"2024-04-15T15:21:11.667227+0000","last_active":"2024-04-15T15:21:16.691111+0000","last_peered":"2024-04-15T15:21:16.691111+0000","last_clean":"2024-04-15T15:21:16.691111+0000","last_became_active":"2024-04-15T15:21:11.663970+0000","last_became_peered":"2024-04-15T15:21:11.663970+0000","last_unstale":"2024-04-15T15:21:16.691111+0000","last_undegraded":"2024-04-15T15:21:16.691111+0000","last_fullsized":"2024-04-15T15:21:16.691111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:02.359534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729911+0000","last_change":"2024-04-15T15:21:11.659069+0000","last_active":"2024-04-15T15:21:16.729911+0000","last_peered":"2024-04-15T15:21:16.729911+0000","last_clean":"2024-04-15T15:21:16.729911+0000","last_became_active":"2024-04-15T15:21:11.658826+0000","last_became_peered":"2024-04-15T15:21:11.658826+0000","last_unstale":"2024-04-15T15:21:16.729911+0000","last_undegraded":"2024-04-15T15:21:16.729911+0000","last_fullsized":"2024-04-15T15:21:16.729911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:34:00.069973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730161+0000","last_change":"2024-04-15T15:21:11.662189+0000","last_active":"2024-04-15T15:21:16.730161+0000","last_peered":"2024-04-15T15:21:16.730161+0000","last_clean":"2024-04-15T15:21:16.730161+0000","last_became_active":"2024-04-15T15:21:11.661850+0000","last_became_peered":"2024-04-15T15:21:11.661850+0000","last_unstale":"2024-04-15T15:21:16.730161+0000","last_undegraded":"2024-04-15T15:21:16.730161+0000","last_fullsized":"2024-04-15T15:21:16.730161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730978+0000","last_change":"2024-04-15T15:21:11.656094+0000","last_active":"2024-04-15T15:21:16.730978+0000","last_peered":"2024-04-15T15:21:16.730978+0000","last_clean":"2024-04-15T15:21:16.730978+0000","last_became_active":"2024-04-15T15:21:11.655585+0000","last_became_peered":"2024-04-15T15:21:11.655585+0000","last_unstale":"2024-04-15T15:21:16.730978+0000","last_undegraded":"2024-04-15T15:21:16.730978+0000","last_fullsized":"2024-04-15T15:21:16.730978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:26.088255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731645+0000","last_change":"2024-04-15T15:21:11.663357+0000","last_active":"2024-04-15T15:21:16.731645+0000","last_peered":"2024-04-15T15:21:16.731645+0000","last_clean":"2024-04-15T15:21:16.731645+0000","last_became_active":"2024-04-15T15:21:11.663051+0000","last_became_peered":"2024-04-15T15:21:11.663051+0000","last_unstale":"2024-04-15T15:21:16.731645+0000","last_undegraded":"2024-04-15T15:21:16.731645+0000","last_fullsized":"2024-04-15T15:21:16.731645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:53.740250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691911+0000","last_change":"2024-04-15T15:21:11.661796+0000","last_active":"2024-04-15T15:21:16.691911+0000","last_peered":"2024-04-15T15:21:16.691911+0000","last_clean":"2024-04-15T15:21:16.691911+0000","last_became_active":"2024-04-15T15:21:11.661288+0000","last_became_peered":"2024-04-15T15:21:11.661288+0000","last_unstale":"2024-04-15T15:21:16.691911+0000","last_undegraded":"2024-04-15T15:21:16.691911+0000","last_fullsized":"2024-04-15T15:21:16.691911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:26.088255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690954+0000","last_change":"2024-04-15T15:21:11.663287+0000","last_active":"2024-04-15T15:21:16.690954+0000","last_peered":"2024-04-15T15:21:16.690954+0000","last_clean":"2024-04-15T15:21:16.690954+0000","last_became_active":"2024-04-15T15:21:11.662684+0000","last_became_peered":"2024-04-15T15:21:11.662684+0000","last_unstale":"2024-04-15T15:21:16.690954+0000","last_undegraded":"2024-04-15T15:21:16.690954+0000","last_fullsized":"2024-04-15T15:21:16.690954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729440+0000","last_change":"2024-04-15T15:21:11.657278+0000","last_active":"2024-04-15T15:21:16.729440+0000","last_peered":"2024-04-15T15:21:16.729440+0000","last_clean":"2024-04-15T15:21:16.729440+0000","last_became_active":"2024-04-15T15:21:11.656878+0000","last_became_peered":"2024-04-15T15:21:11.656878+0000","last_unstale":"2024-04-15T15:21:16.729440+0000","last_undegraded":"2024-04-15T15:21:16.729440+0000","last_fullsized":"2024-04-15T15:21:16.729440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730993+0000","last_change":"2024-04-15T15:21:11.646900+0000","last_active":"2024-04-15T15:21:16.730993+0000","last_peered":"2024-04-15T15:21:16.730993+0000","last_clean":"2024-04-15T15:21:16.730993+0000","last_became_active":"2024-04-15T15:21:11.646504+0000","last_became_peered":"2024-04-15T15:21:11.646504+0000","last_unstale":"2024-04-15T15:21:16.730993+0000","last_undegraded":"2024-04-15T15:21:16.730993+0000","last_fullsized":"2024-04-15T15:21:16.730993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732733+0000","last_change":"2024-04-15T15:21:11.666961+0000","last_active":"2024-04-15T15:21:16.732733+0000","last_peered":"2024-04-15T15:21:16.732733+0000","last_clean":"2024-04-15T15:21:16.732733+0000","last_became_active":"2024-04-15T15:21:11.666715+0000","last_became_peered":"2024-04-15T15:21:11.666715+0000","last_unstale":"2024-04-15T15:21:16.732733+0000","last_undegraded":"2024-04-15T15:21:16.732733+0000","last_fullsized":"2024-04-15T15:21:16.732733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685618+0000","last_change":"2024-04-15T15:21:11.662851+0000","last_active":"2024-04-15T15:21:16.685618+0000","last_peered":"2024-04-15T15:21:16.685618+0000","last_clean":"2024-04-15T15:21:16.685618+0000","last_became_active":"2024-04-15T15:21:11.662536+0000","last_became_peered":"2024-04-15T15:21:11.662536+0000","last_unstale":"2024-04-15T15:21:16.685618+0000","last_undegraded":"2024-04-15T15:21:16.685618+0000","last_fullsized":"2024-04-15T15:21:16.685618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:53.932651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730237+0000","last_change":"2024-04-15T15:21:11.670651+0000","last_active":"2024-04-15T15:21:16.730237+0000","last_peered":"2024-04-15T15:21:16.730237+0000","last_clean":"2024-04-15T15:21:16.730237+0000","last_became_active":"2024-04-15T15:21:11.670354+0000","last_became_peered":"2024-04-15T15:21:11.670354+0000","last_unstale":"2024-04-15T15:21:16.730237+0000","last_undegraded":"2024-04-15T15:21:16.730237+0000","last_fullsized":"2024-04-15T15:21:16.730237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730494+0000","last_change":"2024-04-15T15:21:11.670652+0000","last_active":"2024-04-15T15:21:16.730494+0000","last_peered":"2024-04-15T15:21:16.730494+0000","last_clean":"2024-04-15T15:21:16.730494+0000","last_became_active":"2024-04-15T15:21:11.670396+0000","last_became_peered":"2024-04-15T15:21:11.670396+0000","last_unstale":"2024-04-15T15:21:16.730494+0000","last_undegraded":"2024-04-15T15:21:16.730494+0000","last_fullsized":"2024-04-15T15:21:16.730494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730636+0000","last_change":"2024-04-15T15:21:11.665992+0000","last_active":"2024-04-15T15:21:16.730636+0000","last_peered":"2024-04-15T15:21:16.730636+0000","last_clean":"2024-04-15T15:21:16.730636+0000","last_became_active":"2024-04-15T15:21:11.665531+0000","last_became_peered":"2024-04-15T15:21:11.665531+0000","last_unstale":"2024-04-15T15:21:16.730636+0000","last_undegraded":"2024-04-15T15:21:16.730636+0000","last_fullsized":"2024-04-15T15:21:16.730636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:09.770327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732177+0000","last_change":"2024-04-15T15:21:11.665238+0000","last_active":"2024-04-15T15:21:16.732177+0000","last_peered":"2024-04-15T15:21:16.732177+0000","last_clean":"2024-04-15T15:21:16.732177+0000","last_became_active":"2024-04-15T15:21:11.664931+0000","last_became_peered":"2024-04-15T15:21:11.664931+0000","last_unstale":"2024-04-15T15:21:16.732177+0000","last_undegraded":"2024-04-15T15:21:16.732177+0000","last_fullsized":"2024-04-15T15:21:16.732177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731045+0000","last_change":"2024-04-15T15:21:11.655222+0000","last_active":"2024-04-15T15:21:16.731045+0000","last_peered":"2024-04-15T15:21:16.731045+0000","last_clean":"2024-04-15T15:21:16.731045+0000","last_became_active":"2024-04-15T15:21:11.654826+0000","last_became_peered":"2024-04-15T15:21:11.654826+0000","last_unstale":"2024-04-15T15:21:16.731045+0000","last_undegraded":"2024-04-15T15:21:16.731045+0000","last_fullsized":"2024-04-15T15:21:16.731045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:20.066936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690947+0000","last_change":"2024-04-15T15:21:11.662632+0000","last_active":"2024-04-15T15:21:16.690947+0000","last_peered":"2024-04-15T15:21:16.690947+0000","last_clean":"2024-04-15T15:21:16.690947+0000","last_became_active":"2024-04-15T15:21:11.661632+0000","last_became_peered":"2024-04-15T15:21:11.661632+0000","last_unstale":"2024-04-15T15:21:16.690947+0000","last_undegraded":"2024-04-15T15:21:16.690947+0000","last_fullsized":"2024-04-15T15:21:16.690947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730098+0000","last_change":"2024-04-15T15:21:11.655451+0000","last_active":"2024-04-15T15:21:16.730098+0000","last_peered":"2024-04-15T15:21:16.730098+0000","last_clean":"2024-04-15T15:21:16.730098+0000","last_became_active":"2024-04-15T15:21:11.655138+0000","last_became_peered":"2024-04-15T15:21:11.655138+0000","last_unstale":"2024-04-15T15:21:16.730098+0000","last_undegraded":"2024-04-15T15:21:16.730098+0000","last_fullsized":"2024-04-15T15:21:16.730098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691160+0000","last_change":"2024-04-15T15:21:11.662669+0000","last_active":"2024-04-15T15:21:16.691160+0000","last_peered":"2024-04-15T15:21:16.691160+0000","last_clean":"2024-04-15T15:21:16.691160+0000","last_became_active":"2024-04-15T15:21:11.662385+0000","last_became_peered":"2024-04-15T15:21:11.662385+0000","last_unstale":"2024-04-15T15:21:16.691160+0000","last_undegraded":"2024-04-15T15:21:16.691160+0000","last_fullsized":"2024-04-15T15:21:16.691160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:55.392737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731209+0000","last_change":"2024-04-15T15:21:11.640925+0000","last_active":"2024-04-15T15:21:16.731209+0000","last_peered":"2024-04-15T15:21:16.731209+0000","last_clean":"2024-04-15T15:21:16.731209+0000","last_became_active":"2024-04-15T15:21:11.640618+0000","last_became_peered":"2024-04-15T15:21:11.640618+0000","last_unstale":"2024-04-15T15:21:16.731209+0000","last_undegraded":"2024-04-15T15:21:16.731209+0000","last_fullsized":"2024-04-15T15:21:16.731209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730901+0000","last_change":"2024-04-15T15:21:11.653243+0000","last_active":"2024-04-15T15:21:16.730901+0000","last_peered":"2024-04-15T15:21:16.730901+0000","last_clean":"2024-04-15T15:21:16.730901+0000","last_became_active":"2024-04-15T15:21:11.652962+0000","last_became_peered":"2024-04-15T15:21:11.652962+0000","last_unstale":"2024-04-15T15:21:16.730901+0000","last_undegraded":"2024-04-15T15:21:16.730901+0000","last_fullsized":"2024-04-15T15:21:16.730901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730496+0000","last_change":"2024-04-15T15:21:11.662774+0000","last_active":"2024-04-15T15:21:16.730496+0000","last_peered":"2024-04-15T15:21:16.730496+0000","last_clean":"2024-04-15T15:21:16.730496+0000","last_became_active":"2024-04-15T15:21:11.662488+0000","last_became_peered":"2024-04-15T15:21:11.662488+0000","last_unstale":"2024-04-15T15:21:16.730496+0000","last_undegraded":"2024-04-15T15:21:16.730496+0000","last_fullsized":"2024-04-15T15:21:16.730496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:09.923283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:21:37.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:21:37.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.722+0000 7fc86c972640 1 Processor -- start 2024-04-15T15:21:37.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.722+0000 7fc86c972640 1 -- start start 2024-04-15T15:21:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc86c972640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc86815ea40 0x7fc8680b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc86c972640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8680d3480 0x7fc8680b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc86c972640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8680cff70 con 0x7fc86815ee20 2024-04-15T15:21:37.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc86c972640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc8680d0250 con 0x7fc86815ea40 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc86c972640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc8680d0530 con 0x7fc8680d3480 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc86659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc86815ea40 0x7fc8680b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc85ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc86815ee20 0x7fc8680b6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56414/0 (socket says 172.21.15.12:56414) 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc86659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc86815ea40 0x7fc8680b5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46938/0 (socket says 172.21.15.12:46938) 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.723+0000 7fc85ffff640 1 -- 172.21.15.12:0/1177805962 learned_addr learned my addr 172.21.15.12:0/1177805962 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.724+0000 7fc866d9d640 1 --2- 172.21.15.12:0/1177805962 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8680d3480 0x7fc8680b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.724+0000 7fc86659c640 1 -- 172.21.15.12:0/1177805962 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8680d3480 msgr2=0x7fc8680b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.724+0000 7fc86659c640 1 --2- 172.21.15.12:0/1177805962 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8680d3480 0x7fc8680b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.724+0000 7fc86659c640 1 -- 172.21.15.12:0/1177805962 >> v1:172.21.15.12:6789/0 conn(0x7fc86815ee20 legacy=0x7fc8680b6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.724+0000 7fc86659c640 1 -- 172.21.15.12:0/1177805962 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8680d0810 con 0x7fc86815ea40 2024-04-15T15:21:37.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.724+0000 7fc865d9b640 1 -- 172.21.15.12:0/1177805962 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108885675 0 0) 0x7fc8680cff70 con 0x7fc86815ee20 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.724+0000 7fc86659c640 1 --2- 172.21.15.12:0/1177805962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ea40 0x7fc8680b5b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc8580097a0 tx=0x7fc85802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1956e0e2c992a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc865d9b640 1 -- 172.21.15.12:0/1177805962 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858034070 con 0x7fc86815ea40 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc865d9b640 1 -- 172.21.15.12:0/1177805962 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc858030610 con 0x7fc86815ea40 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc865d9b640 1 -- 172.21.15.12:0/1177805962 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8580309e0 con 0x7fc86815ea40 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc86c972640 1 -- 172.21.15.12:0/1177805962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ea40 msgr2=0x7fc8680b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc86c972640 1 --2- 172.21.15.12:0/1177805962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ea40 0x7fc8680b5b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc8580097a0 tx=0x7fc85802b960 comp rx=0 tx=0).stop 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc86c972640 1 -- 172.21.15.12:0/1177805962 shutdown_connections 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc86c972640 1 --2- 172.21.15.12:0/1177805962 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc8680d3480 0x7fc8680b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc86c972640 1 --2- 172.21.15.12:0/1177805962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ea40 0x7fc8680b5b40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc86c972640 1 -- 172.21.15.12:0/1177805962 >> 172.21.15.12:0/1177805962 conn(0x7fc86808bf60 msgr2=0x7fc8680d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc86c972640 1 -- 172.21.15.12:0/1177805962 shutdown_connections 2024-04-15T15:21:37.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.725+0000 7fc86c972640 1 -- 172.21.15.12:0/1177805962 wait complete. 2024-04-15T15:21:37.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.726+0000 7fc86c972640 1 Processor -- start 2024-04-15T15:21:37.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.726+0000 7fc86c972640 1 -- start start 2024-04-15T15:21:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc86c972640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8680d3480 0x7fc8680b99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc86c972640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc86815ea40 0x7fc8681f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc86c972640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ee20 0x7fc8681615b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc86c972640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc8680b3a40 con 0x7fc86815ee20 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc86c972640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc8680b38c0 con 0x7fc8680d3480 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc86c972640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc8680b6790 con 0x7fc86815ea40 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc866d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ee20 0x7fc8681615b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc866d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ee20 0x7fc8681615b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46950/0 (socket says 172.21.15.12:46950) 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.727+0000 7fc866d9d640 1 -- 172.21.15.12:0/3912213893 learned_addr learned my addr 172.21.15.12:0/3912213893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc86659c640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8680d3480 0x7fc8680b99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc866d9d640 1 -- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc86815ea40 msgr2=0x7fc8681f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc85ffff640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc86815ea40 0x7fc8681f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc866d9d640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc86815ea40 0x7fc8681f7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc866d9d640 1 -- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8680d3480 msgr2=0x7fc8680b99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc866d9d640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8680d3480 0x7fc8680b99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc866d9d640 1 -- 172.21.15.12:0/3912213893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8680b73c0 con 0x7fc86815ee20 2024-04-15T15:21:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc86659c640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8680d3480 0x7fc8680b99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc866d9d640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ee20 0x7fc8681615b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc860007e40 tx=0x7fc86000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc85effd640 1 -- 172.21.15.12:0/3912213893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860014620 con 0x7fc86815ee20 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc85effd640 1 -- 172.21.15.12:0/3912213893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc860019070 con 0x7fc86815ee20 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8680b76a0 con 0x7fc86815ee20 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc85effd640 1 -- 172.21.15.12:0/3912213893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860014a20 con 0x7fc86815ee20 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.728+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8680b7c60 con 0x7fc86815ee20 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.729+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8680689e0 con 0x7fc86815ee20 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.729+0000 7fc85effd640 1 -- 172.21.15.12:0/3912213893 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc860014bc0 con 0x7fc86815ee20 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.730+0000 7fc85effd640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc84806c8f0 0x7fc84806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.730+0000 7fc85effd640 1 -- 172.21.15.12:0/3912213893 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc86008c710 con 0x7fc86815ee20 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.730+0000 7fc86659c640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc84806c8f0 0x7fc84806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.730+0000 7fc86659c640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc84806c8f0 0x7fc84806edb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc858036000 tx=0x7fc85803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.734+0000 7fc85effd640 1 -- 172.21.15.12:0/3912213893 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc86005a890 con 0x7fc86815ee20 2024-04-15T15:21:37.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.919+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8680b3140 con 0x7fc84806c8f0 2024-04-15T15:21:37.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.927+0000 7fc85effd640 1 -- 172.21.15.12:0/3912213893 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7fc8680b3140 con 0x7fc84806c8f0 2024-04-15T15:21:37.932 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:37.934 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:21:37.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.930+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc84806c8f0 msgr2=0x7fc84806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.930+0000 7fc86c972640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc84806c8f0 0x7fc84806edb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc858036000 tx=0x7fc85803b040 comp rx=0 tx=0).stop 2024-04-15T15:21:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ee20 msgr2=0x7fc8681615b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ee20 0x7fc8681615b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc860007e40 tx=0x7fc86000e4b0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 shutdown_connections 2024-04-15T15:21:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc86815ee20 0x7fc8681615b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc86815ea40 0x7fc8681f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc84806c8f0 0x7fc84806edb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 --2- 172.21.15.12:0/3912213893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8680d3480 0x7fc8680b99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 >> 172.21.15.12:0/3912213893 conn(0x7fc86808bf60 msgr2=0x7fc86815c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 shutdown_connections 2024-04-15T15:21:37.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:37.931+0000 7fc86c972640 1 -- 172.21.15.12:0/3912213893 wait complete. 2024-04-15T15:21:37.943 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-15T15:21:36.593785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236140,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738580,"statfs":{"total":773094113280,"available":772852305920,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003138"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731482+0000","last_change":"2024-04-15T15:21:10.650644+0000","last_active":"2024-04-15T15:21:16.731482+0000","last_peered":"2024-04-15T15:21:16.731482+0000","last_clean":"2024-04-15T15:21:16.731482+0000","last_became_active":"2024-04-15T15:21:10.650315+0000","last_became_peered":"2024-04-15T15:21:10.650315+0000","last_unstale":"2024-04-15T15:21:16.731482+0000","last_undegraded":"2024-04-15T15:21:16.731482+0000","last_fullsized":"2024-04-15T15:21:16.731482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:27.659449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732827+0000","last_change":"2024-04-15T15:21:11.667138+0000","last_active":"2024-04-15T15:21:16.732827+0000","last_peered":"2024-04-15T15:21:16.732827+0000","last_clean":"2024-04-15T15:21:16.732827+0000","last_became_active":"2024-04-15T15:21:11.666799+0000","last_became_peered":"2024-04-15T15:21:11.666799+0000","last_unstale":"2024-04-15T15:21:16.732827+0000","last_undegraded":"2024-04-15T15:21:16.732827+0000","last_fullsized":"2024-04-15T15:21:16.732827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:24.468426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732274+0000","last_change":"2024-04-15T15:21:15.901408+0000","last_active":"2024-04-15T15:21:16.732274+0000","last_peered":"2024-04-15T15:21:16.732274+0000","last_clean":"2024-04-15T15:21:16.732274+0000","last_became_active":"2024-04-15T15:21:15.901032+0000","last_became_peered":"2024-04-15T15:21:15.901032+0000","last_unstale":"2024-04-15T15:21:16.732274+0000","last_undegraded":"2024-04-15T15:21:16.732274+0000","last_fullsized":"2024-04-15T15:21:16.732274+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:34.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733027+0000","last_change":"2024-04-15T15:21:10.636248+0000","last_active":"2024-04-15T15:21:16.733027+0000","last_peered":"2024-04-15T15:21:16.733027+0000","last_clean":"2024-04-15T15:21:16.733027+0000","last_became_active":"2024-04-15T15:21:10.635963+0000","last_became_peered":"2024-04-15T15:21:10.635963+0000","last_unstale":"2024-04-15T15:21:16.733027+0000","last_undegraded":"2024-04-15T15:21:16.733027+0000","last_fullsized":"2024-04-15T15:21:16.733027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:27.659449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731287+0000","last_change":"2024-04-15T15:21:11.665760+0000","last_active":"2024-04-15T15:21:16.731287+0000","last_peered":"2024-04-15T15:21:16.731287+0000","last_clean":"2024-04-15T15:21:16.731287+0000","last_became_active":"2024-04-15T15:21:11.665444+0000","last_became_peered":"2024-04-15T15:21:11.665444+0000","last_unstale":"2024-04-15T15:21:16.731287+0000","last_undegraded":"2024-04-15T15:21:16.731287+0000","last_fullsized":"2024-04-15T15:21:16.731287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.684953+0000","last_change":"2024-04-15T15:21:15.699988+0000","last_active":"2024-04-15T15:21:16.684953+0000","last_peered":"2024-04-15T15:21:16.684953+0000","last_clean":"2024-04-15T15:21:16.684953+0000","last_became_active":"2024-04-15T15:21:15.699333+0000","last_became_peered":"2024-04-15T15:21:15.699333+0000","last_unstale":"2024-04-15T15:21:16.684953+0000","last_undegraded":"2024-04-15T15:21:16.684953+0000","last_fullsized":"2024-04-15T15:21:16.684953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:26.928706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730928+0000","last_change":"2024-04-15T15:21:10.649466+0000","last_active":"2024-04-15T15:21:16.730928+0000","last_peered":"2024-04-15T15:21:16.730928+0000","last_clean":"2024-04-15T15:21:16.730928+0000","last_became_active":"2024-04-15T15:21:10.649182+0000","last_became_peered":"2024-04-15T15:21:10.649182+0000","last_unstale":"2024-04-15T15:21:16.730928+0000","last_undegraded":"2024-04-15T15:21:16.730928+0000","last_fullsized":"2024-04-15T15:21:16.730928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:01:16.245697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691301+0000","last_change":"2024-04-15T15:21:11.667011+0000","last_active":"2024-04-15T15:21:16.691301+0000","last_peered":"2024-04-15T15:21:16.691301+0000","last_clean":"2024-04-15T15:21:16.691301+0000","last_became_active":"2024-04-15T15:21:11.663992+0000","last_became_peered":"2024-04-15T15:21:11.663992+0000","last_unstale":"2024-04-15T15:21:16.691301+0000","last_undegraded":"2024-04-15T15:21:16.691301+0000","last_fullsized":"2024-04-15T15:21:16.691301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685332+0000","last_change":"2024-04-15T15:21:15.707372+0000","last_active":"2024-04-15T15:21:16.685332+0000","last_peered":"2024-04-15T15:21:16.685332+0000","last_clean":"2024-04-15T15:21:16.685332+0000","last_became_active":"2024-04-15T15:21:15.706978+0000","last_became_peered":"2024-04-15T15:21:15.706978+0000","last_unstale":"2024-04-15T15:21:16.685332+0000","last_undegraded":"2024-04-15T15:21:16.685332+0000","last_fullsized":"2024-04-15T15:21:16.685332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:32.327246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730471+0000","last_change":"2024-04-15T15:21:10.652340+0000","last_active":"2024-04-15T15:21:16.730471+0000","last_peered":"2024-04-15T15:21:16.730471+0000","last_clean":"2024-04-15T15:21:16.730471+0000","last_became_active":"2024-04-15T15:21:10.652016+0000","last_became_peered":"2024-04-15T15:21:10.652016+0000","last_unstale":"2024-04-15T15:21:16.730471+0000","last_undegraded":"2024-04-15T15:21:16.730471+0000","last_fullsized":"2024-04-15T15:21:16.730471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:54:08.208968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691646+0000","last_change":"2024-04-15T15:21:11.663781+0000","last_active":"2024-04-15T15:21:16.691646+0000","last_peered":"2024-04-15T15:21:16.691646+0000","last_clean":"2024-04-15T15:21:16.691646+0000","last_became_active":"2024-04-15T15:21:11.663232+0000","last_became_peered":"2024-04-15T15:21:11.663232+0000","last_unstale":"2024-04-15T15:21:16.691646+0000","last_undegraded":"2024-04-15T15:21:16.691646+0000","last_fullsized":"2024-04-15T15:21:16.691646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.568336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685080+0000","last_change":"2024-04-15T15:21:15.695846+0000","last_active":"2024-04-15T15:21:16.685080+0000","last_peered":"2024-04-15T15:21:16.685080+0000","last_clean":"2024-04-15T15:21:16.685080+0000","last_became_active":"2024-04-15T15:21:15.695426+0000","last_became_peered":"2024-04-15T15:21:15.695426+0000","last_unstale":"2024-04-15T15:21:16.685080+0000","last_undegraded":"2024-04-15T15:21:16.685080+0000","last_fullsized":"2024-04-15T15:21:16.685080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:28.518948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729851+0000","last_change":"2024-04-15T15:21:11.662485+0000","last_active":"2024-04-15T15:21:16.729851+0000","last_peered":"2024-04-15T15:21:16.729851+0000","last_clean":"2024-04-15T15:21:16.729851+0000","last_became_active":"2024-04-15T15:21:11.662183+0000","last_became_peered":"2024-04-15T15:21:11.662183+0000","last_unstale":"2024-04-15T15:21:16.729851+0000","last_undegraded":"2024-04-15T15:21:16.729851+0000","last_fullsized":"2024-04-15T15:21:16.729851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730547+0000","last_change":"2024-04-15T15:21:10.648295+0000","last_active":"2024-04-15T15:21:16.730547+0000","last_peered":"2024-04-15T15:21:16.730547+0000","last_clean":"2024-04-15T15:21:16.730547+0000","last_became_active":"2024-04-15T15:21:10.647985+0000","last_became_peered":"2024-04-15T15:21:10.647985+0000","last_unstale":"2024-04-15T15:21:16.730547+0000","last_undegraded":"2024-04-15T15:21:16.730547+0000","last_fullsized":"2024-04-15T15:21:16.730547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:52.730839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685664+0000","last_change":"2024-04-15T15:21:15.699611+0000","last_active":"2024-04-15T15:21:16.685664+0000","last_peered":"2024-04-15T15:21:16.685664+0000","last_clean":"2024-04-15T15:21:16.685664+0000","last_became_active":"2024-04-15T15:21:15.699129+0000","last_became_peered":"2024-04-15T15:21:15.699129+0000","last_unstale":"2024-04-15T15:21:16.685664+0000","last_undegraded":"2024-04-15T15:21:16.685664+0000","last_fullsized":"2024-04-15T15:21:16.685664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732647+0000","last_change":"2024-04-15T15:21:11.644620+0000","last_active":"2024-04-15T15:21:16.732647+0000","last_peered":"2024-04-15T15:21:16.732647+0000","last_clean":"2024-04-15T15:21:16.732647+0000","last_became_active":"2024-04-15T15:21:11.644273+0000","last_became_peered":"2024-04-15T15:21:11.644273+0000","last_unstale":"2024-04-15T15:21:16.732647+0000","last_undegraded":"2024-04-15T15:21:16.732647+0000","last_fullsized":"2024-04-15T15:21:16.732647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730443+0000","last_change":"2024-04-15T15:21:10.636457+0000","last_active":"2024-04-15T15:21:16.730443+0000","last_peered":"2024-04-15T15:21:16.730443+0000","last_clean":"2024-04-15T15:21:16.730443+0000","last_became_active":"2024-04-15T15:21:10.636143+0000","last_became_peered":"2024-04-15T15:21:10.636143+0000","last_unstale":"2024-04-15T15:21:16.730443+0000","last_undegraded":"2024-04-15T15:21:16.730443+0000","last_fullsized":"2024-04-15T15:21:16.730443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685684+0000","last_change":"2024-04-15T15:21:15.703570+0000","last_active":"2024-04-15T15:21:16.685684+0000","last_peered":"2024-04-15T15:21:16.685684+0000","last_clean":"2024-04-15T15:21:16.685684+0000","last_became_active":"2024-04-15T15:21:15.703066+0000","last_became_peered":"2024-04-15T15:21:15.703066+0000","last_unstale":"2024-04-15T15:21:16.685684+0000","last_undegraded":"2024-04-15T15:21:16.685684+0000","last_fullsized":"2024-04-15T15:21:16.685684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732393+0000","last_change":"2024-04-15T15:21:11.646745+0000","last_active":"2024-04-15T15:21:16.732393+0000","last_peered":"2024-04-15T15:21:16.732393+0000","last_clean":"2024-04-15T15:21:16.732393+0000","last_became_active":"2024-04-15T15:21:11.646375+0000","last_became_peered":"2024-04-15T15:21:11.646375+0000","last_unstale":"2024-04-15T15:21:16.732393+0000","last_undegraded":"2024-04-15T15:21:16.732393+0000","last_fullsized":"2024-04-15T15:21:16.732393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729475+0000","last_change":"2024-04-15T15:21:10.642462+0000","last_active":"2024-04-15T15:21:16.729475+0000","last_peered":"2024-04-15T15:21:16.729475+0000","last_clean":"2024-04-15T15:21:16.729475+0000","last_became_active":"2024-04-15T15:21:10.642168+0000","last_became_peered":"2024-04-15T15:21:10.642168+0000","last_unstale":"2024-04-15T15:21:16.729475+0000","last_undegraded":"2024-04-15T15:21:16.729475+0000","last_fullsized":"2024-04-15T15:21:16.729475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731042+0000","last_change":"2024-04-15T15:21:15.901600+0000","last_active":"2024-04-15T15:21:16.731042+0000","last_peered":"2024-04-15T15:21:16.731042+0000","last_clean":"2024-04-15T15:21:16.731042+0000","last_became_active":"2024-04-15T15:21:15.901233+0000","last_became_peered":"2024-04-15T15:21:15.901233+0000","last_unstale":"2024-04-15T15:21:16.731042+0000","last_undegraded":"2024-04-15T15:21:16.731042+0000","last_fullsized":"2024-04-15T15:21:16.731042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:30.138777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685208+0000","last_change":"2024-04-15T15:21:11.648506+0000","last_active":"2024-04-15T15:21:16.685208+0000","last_peered":"2024-04-15T15:21:16.685208+0000","last_clean":"2024-04-15T15:21:16.685208+0000","last_became_active":"2024-04-15T15:21:11.647972+0000","last_became_peered":"2024-04-15T15:21:11.647972+0000","last_unstale":"2024-04-15T15:21:16.685208+0000","last_undegraded":"2024-04-15T15:21:16.685208+0000","last_fullsized":"2024-04-15T15:21:16.685208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:09.853761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729694+0000","last_change":"2024-04-15T15:21:10.646955+0000","last_active":"2024-04-15T15:21:16.729694+0000","last_peered":"2024-04-15T15:21:16.729694+0000","last_clean":"2024-04-15T15:21:16.729694+0000","last_became_active":"2024-04-15T15:21:10.646308+0000","last_became_peered":"2024-04-15T15:21:10.646308+0000","last_unstale":"2024-04-15T15:21:16.729694+0000","last_undegraded":"2024-04-15T15:21:16.729694+0000","last_fullsized":"2024-04-15T15:21:16.729694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685568+0000","last_change":"2024-04-15T15:21:15.699814+0000","last_active":"2024-04-15T15:21:16.685568+0000","last_peered":"2024-04-15T15:21:16.685568+0000","last_clean":"2024-04-15T15:21:16.685568+0000","last_became_active":"2024-04-15T15:21:15.699341+0000","last_became_peered":"2024-04-15T15:21:15.699341+0000","last_unstale":"2024-04-15T15:21:16.685568+0000","last_undegraded":"2024-04-15T15:21:16.685568+0000","last_fullsized":"2024-04-15T15:21:16.685568+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:51.618858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731527+0000","last_change":"2024-04-15T15:21:11.646950+0000","last_active":"2024-04-15T15:21:16.731527+0000","last_peered":"2024-04-15T15:21:16.731527+0000","last_clean":"2024-04-15T15:21:16.731527+0000","last_became_active":"2024-04-15T15:21:11.646633+0000","last_became_peered":"2024-04-15T15:21:11.646633+0000","last_unstale":"2024-04-15T15:21:16.731527+0000","last_undegraded":"2024-04-15T15:21:16.731527+0000","last_fullsized":"2024-04-15T15:21:16.731527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:56.796018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730414+0000","last_change":"2024-04-15T15:21:10.646693+0000","last_active":"2024-04-15T15:21:16.730414+0000","last_peered":"2024-04-15T15:21:16.730414+0000","last_clean":"2024-04-15T15:21:16.730414+0000","last_became_active":"2024-04-15T15:21:10.646328+0000","last_became_peered":"2024-04-15T15:21:10.646328+0000","last_unstale":"2024-04-15T15:21:16.730414+0000","last_undegraded":"2024-04-15T15:21:16.730414+0000","last_fullsized":"2024-04-15T15:21:16.730414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685127+0000","last_change":"2024-04-15T15:21:15.902524+0000","last_active":"2024-04-15T15:21:16.685127+0000","last_peered":"2024-04-15T15:21:16.685127+0000","last_clean":"2024-04-15T15:21:16.685127+0000","last_became_active":"2024-04-15T15:21:15.901936+0000","last_became_peered":"2024-04-15T15:21:15.901936+0000","last_unstale":"2024-04-15T15:21:16.685127+0000","last_undegraded":"2024-04-15T15:21:16.685127+0000","last_fullsized":"2024-04-15T15:21:16.685127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:59.443259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733036+0000","last_change":"2024-04-15T15:21:11.657367+0000","last_active":"2024-04-15T15:21:16.733036+0000","last_peered":"2024-04-15T15:21:16.733036+0000","last_clean":"2024-04-15T15:21:16.733036+0000","last_became_active":"2024-04-15T15:21:11.657002+0000","last_became_peered":"2024-04-15T15:21:11.657002+0000","last_unstale":"2024-04-15T15:21:16.733036+0000","last_undegraded":"2024-04-15T15:21:16.733036+0000","last_fullsized":"2024-04-15T15:21:16.733036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:02.359534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731010+0000","last_change":"2024-04-15T15:21:10.644925+0000","last_active":"2024-04-15T15:21:16.731010+0000","last_peered":"2024-04-15T15:21:16.731010+0000","last_clean":"2024-04-15T15:21:16.731010+0000","last_became_active":"2024-04-15T15:21:10.644639+0000","last_became_peered":"2024-04-15T15:21:10.644639+0000","last_unstale":"2024-04-15T15:21:16.731010+0000","last_undegraded":"2024-04-15T15:21:16.731010+0000","last_fullsized":"2024-04-15T15:21:16.731010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732150+0000","last_change":"2024-04-15T15:21:15.902755+0000","last_active":"2024-04-15T15:21:16.732150+0000","last_peered":"2024-04-15T15:21:16.732150+0000","last_clean":"2024-04-15T15:21:16.732150+0000","last_became_active":"2024-04-15T15:21:15.902338+0000","last_became_peered":"2024-04-15T15:21:15.902338+0000","last_unstale":"2024-04-15T15:21:16.732150+0000","last_undegraded":"2024-04-15T15:21:16.732150+0000","last_fullsized":"2024-04-15T15:21:16.732150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:13.820849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730647+0000","last_change":"2024-04-15T15:21:11.654427+0000","last_active":"2024-04-15T15:21:16.730647+0000","last_peered":"2024-04-15T15:21:16.730647+0000","last_clean":"2024-04-15T15:21:16.730647+0000","last_became_active":"2024-04-15T15:21:11.654037+0000","last_became_peered":"2024-04-15T15:21:11.654037+0000","last_unstale":"2024-04-15T15:21:16.730647+0000","last_undegraded":"2024-04-15T15:21:16.730647+0000","last_fullsized":"2024-04-15T15:21:16.730647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731058+0000","last_change":"2024-04-15T15:21:10.649235+0000","last_active":"2024-04-15T15:21:16.731058+0000","last_peered":"2024-04-15T15:21:16.731058+0000","last_clean":"2024-04-15T15:21:16.731058+0000","last_became_active":"2024-04-15T15:21:10.648935+0000","last_became_peered":"2024-04-15T15:21:10.648935+0000","last_unstale":"2024-04-15T15:21:16.731058+0000","last_undegraded":"2024-04-15T15:21:16.731058+0000","last_fullsized":"2024-04-15T15:21:16.731058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:55.786608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729854+0000","last_change":"2024-04-15T15:21:15.899286+0000","last_active":"2024-04-15T15:21:16.729854+0000","last_peered":"2024-04-15T15:21:16.729854+0000","last_clean":"2024-04-15T15:21:16.729854+0000","last_became_active":"2024-04-15T15:21:15.898902+0000","last_became_peered":"2024-04-15T15:21:15.898902+0000","last_unstale":"2024-04-15T15:21:16.729854+0000","last_undegraded":"2024-04-15T15:21:16.729854+0000","last_fullsized":"2024-04-15T15:21:16.729854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691740+0000","last_change":"2024-04-15T15:21:11.661691+0000","last_active":"2024-04-15T15:21:16.691740+0000","last_peered":"2024-04-15T15:21:16.691740+0000","last_clean":"2024-04-15T15:21:16.691740+0000","last_became_active":"2024-04-15T15:21:11.661289+0000","last_became_peered":"2024-04-15T15:21:11.661289+0000","last_unstale":"2024-04-15T15:21:16.691740+0000","last_undegraded":"2024-04-15T15:21:16.691740+0000","last_fullsized":"2024-04-15T15:21:16.691740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:08.032185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730825+0000","last_change":"2024-04-15T15:21:10.637826+0000","last_active":"2024-04-15T15:21:16.730825+0000","last_peered":"2024-04-15T15:21:16.730825+0000","last_clean":"2024-04-15T15:21:16.730825+0000","last_became_active":"2024-04-15T15:21:10.637566+0000","last_became_peered":"2024-04-15T15:21:10.637566+0000","last_unstale":"2024-04-15T15:21:16.730825+0000","last_undegraded":"2024-04-15T15:21:16.730825+0000","last_fullsized":"2024-04-15T15:21:16.730825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:31.743027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729641+0000","last_change":"2024-04-15T15:21:15.707987+0000","last_active":"2024-04-15T15:21:16.729641+0000","last_peered":"2024-04-15T15:21:16.729641+0000","last_clean":"2024-04-15T15:21:16.729641+0000","last_became_active":"2024-04-15T15:21:15.707472+0000","last_became_peered":"2024-04-15T15:21:15.707472+0000","last_unstale":"2024-04-15T15:21:16.729641+0000","last_undegraded":"2024-04-15T15:21:16.729641+0000","last_fullsized":"2024-04-15T15:21:16.729641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685621+0000","last_change":"2024-04-15T15:21:11.644409+0000","last_active":"2024-04-15T15:21:16.685621+0000","last_peered":"2024-04-15T15:21:16.685621+0000","last_clean":"2024-04-15T15:21:16.685621+0000","last_became_active":"2024-04-15T15:21:11.644095+0000","last_became_peered":"2024-04-15T15:21:11.644095+0000","last_unstale":"2024-04-15T15:21:16.685621+0000","last_undegraded":"2024-04-15T15:21:16.685621+0000","last_fullsized":"2024-04-15T15:21:16.685621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:32.752437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691377+0000","last_change":"2024-04-15T15:21:10.646053+0000","last_active":"2024-04-15T15:21:16.691377+0000","last_peered":"2024-04-15T15:21:16.691377+0000","last_clean":"2024-04-15T15:21:16.691377+0000","last_became_active":"2024-04-15T15:21:10.645737+0000","last_became_peered":"2024-04-15T15:21:10.645737+0000","last_unstale":"2024-04-15T15:21:16.691377+0000","last_undegraded":"2024-04-15T15:21:16.691377+0000","last_fullsized":"2024-04-15T15:21:16.691377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:37.462770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730648+0000","last_change":"2024-04-15T15:21:15.699348+0000","last_active":"2024-04-15T15:21:16.730648+0000","last_peered":"2024-04-15T15:21:16.730648+0000","last_clean":"2024-04-15T15:21:16.730648+0000","last_became_active":"2024-04-15T15:21:15.698962+0000","last_became_peered":"2024-04-15T15:21:15.698962+0000","last_unstale":"2024-04-15T15:21:16.730648+0000","last_undegraded":"2024-04-15T15:21:16.730648+0000","last_fullsized":"2024-04-15T15:21:16.730648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:35.384335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729484+0000","last_change":"2024-04-15T15:21:11.653613+0000","last_active":"2024-04-15T15:21:16.729484+0000","last_peered":"2024-04-15T15:21:16.729484+0000","last_clean":"2024-04-15T15:21:16.729484+0000","last_became_active":"2024-04-15T15:21:11.653306+0000","last_became_peered":"2024-04-15T15:21:11.653306+0000","last_unstale":"2024-04-15T15:21:16.729484+0000","last_undegraded":"2024-04-15T15:21:16.729484+0000","last_fullsized":"2024-04-15T15:21:16.729484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731231+0000","last_change":"2024-04-15T15:21:10.636526+0000","last_active":"2024-04-15T15:21:16.731231+0000","last_peered":"2024-04-15T15:21:16.731231+0000","last_clean":"2024-04-15T15:21:16.731231+0000","last_became_active":"2024-04-15T15:21:10.636211+0000","last_became_peered":"2024-04-15T15:21:10.636211+0000","last_unstale":"2024-04-15T15:21:16.731231+0000","last_undegraded":"2024-04-15T15:21:16.731231+0000","last_fullsized":"2024-04-15T15:21:16.731231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731153+0000","last_change":"2024-04-15T15:21:15.706536+0000","last_active":"2024-04-15T15:21:16.731153+0000","last_peered":"2024-04-15T15:21:16.731153+0000","last_clean":"2024-04-15T15:21:16.731153+0000","last_became_active":"2024-04-15T15:21:15.706130+0000","last_became_peered":"2024-04-15T15:21:15.706130+0000","last_unstale":"2024-04-15T15:21:16.731153+0000","last_undegraded":"2024-04-15T15:21:16.731153+0000","last_fullsized":"2024-04-15T15:21:16.731153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:35.371338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729702+0000","last_change":"2024-04-15T15:21:11.649636+0000","last_active":"2024-04-15T15:21:16.729702+0000","last_peered":"2024-04-15T15:21:16.729702+0000","last_clean":"2024-04-15T15:21:16.729702+0000","last_became_active":"2024-04-15T15:21:11.649334+0000","last_became_peered":"2024-04-15T15:21:11.649334+0000","last_unstale":"2024-04-15T15:21:16.729702+0000","last_undegraded":"2024-04-15T15:21:16.729702+0000","last_fullsized":"2024-04-15T15:21:16.729702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:01:17.255107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730694+0000","last_change":"2024-04-15T15:21:10.647209+0000","last_active":"2024-04-15T15:21:16.730694+0000","last_peered":"2024-04-15T15:21:16.730694+0000","last_clean":"2024-04-15T15:21:16.730694+0000","last_became_active":"2024-04-15T15:21:10.646650+0000","last_became_peered":"2024-04-15T15:21:10.646650+0000","last_unstale":"2024-04-15T15:21:16.730694+0000","last_undegraded":"2024-04-15T15:21:16.730694+0000","last_fullsized":"2024-04-15T15:21:16.730694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:31.743027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731537+0000","last_change":"2024-04-15T15:21:15.901283+0000","last_active":"2024-04-15T15:21:16.731537+0000","last_peered":"2024-04-15T15:21:16.731537+0000","last_clean":"2024-04-15T15:21:16.731537+0000","last_became_active":"2024-04-15T15:21:15.900931+0000","last_became_peered":"2024-04-15T15:21:15.900931+0000","last_unstale":"2024-04-15T15:21:16.731537+0000","last_undegraded":"2024-04-15T15:21:16.731537+0000","last_fullsized":"2024-04-15T15:21:16.731537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:35.371338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732331+0000","last_change":"2024-04-15T15:21:11.664990+0000","last_active":"2024-04-15T15:21:16.732331+0000","last_peered":"2024-04-15T15:21:16.732331+0000","last_clean":"2024-04-15T15:21:16.732331+0000","last_became_active":"2024-04-15T15:21:11.664706+0000","last_became_peered":"2024-04-15T15:21:11.664706+0000","last_unstale":"2024-04-15T15:21:16.732331+0000","last_undegraded":"2024-04-15T15:21:16.732331+0000","last_fullsized":"2024-04-15T15:21:16.732331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691196+0000","last_change":"2024-04-15T15:21:10.645516+0000","last_active":"2024-04-15T15:21:16.691196+0000","last_peered":"2024-04-15T15:21:16.691196+0000","last_clean":"2024-04-15T15:21:16.691196+0000","last_became_active":"2024-04-15T15:21:10.645191+0000","last_became_peered":"2024-04-15T15:21:10.645191+0000","last_unstale":"2024-04-15T15:21:16.691196+0000","last_undegraded":"2024-04-15T15:21:16.691196+0000","last_fullsized":"2024-04-15T15:21:16.691196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730801+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:21:16.730801+0000","last_peered":"2024-04-15T15:21:16.730801+0000","last_clean":"2024-04-15T15:21:16.730801+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:21:16.730801+0000","last_undegraded":"2024-04-15T15:21:16.730801+0000","last_fullsized":"2024-04-15T15:21:16.730801+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730072+0000","last_change":"2024-04-15T15:21:15.901627+0000","last_active":"2024-04-15T15:21:16.730072+0000","last_peered":"2024-04-15T15:21:16.730072+0000","last_clean":"2024-04-15T15:21:16.730072+0000","last_became_active":"2024-04-15T15:21:15.901132+0000","last_became_peered":"2024-04-15T15:21:15.901132+0000","last_unstale":"2024-04-15T15:21:16.730072+0000","last_undegraded":"2024-04-15T15:21:16.730072+0000","last_fullsized":"2024-04-15T15:21:16.730072+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:12.082706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730781+0000","last_change":"2024-04-15T15:21:11.665709+0000","last_active":"2024-04-15T15:21:16.730781+0000","last_peered":"2024-04-15T15:21:16.730781+0000","last_clean":"2024-04-15T15:21:16.730781+0000","last_became_active":"2024-04-15T15:21:11.665254+0000","last_became_peered":"2024-04-15T15:21:11.665254+0000","last_unstale":"2024-04-15T15:21:16.730781+0000","last_undegraded":"2024-04-15T15:21:16.730781+0000","last_fullsized":"2024-04-15T15:21:16.730781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:31.333813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730984+0000","last_change":"2024-04-15T15:21:10.644131+0000","last_active":"2024-04-15T15:21:16.730984+0000","last_peered":"2024-04-15T15:21:16.730984+0000","last_clean":"2024-04-15T15:21:16.730984+0000","last_became_active":"2024-04-15T15:21:10.643844+0000","last_became_peered":"2024-04-15T15:21:10.643844+0000","last_unstale":"2024-04-15T15:21:16.730984+0000","last_undegraded":"2024-04-15T15:21:16.730984+0000","last_fullsized":"2024-04-15T15:21:16.730984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:16:25.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729900+0000","last_change":"2024-04-15T15:21:15.899284+0000","last_active":"2024-04-15T15:21:16.729900+0000","last_peered":"2024-04-15T15:21:16.729900+0000","last_clean":"2024-04-15T15:21:16.729900+0000","last_became_active":"2024-04-15T15:21:15.898822+0000","last_became_peered":"2024-04-15T15:21:15.898822+0000","last_unstale":"2024-04-15T15:21:16.729900+0000","last_undegraded":"2024-04-15T15:21:16.729900+0000","last_fullsized":"2024-04-15T15:21:16.729900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:24.117458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691307+0000","last_change":"2024-04-15T15:21:11.663221+0000","last_active":"2024-04-15T15:21:16.691307+0000","last_peered":"2024-04-15T15:21:16.691307+0000","last_clean":"2024-04-15T15:21:16.691307+0000","last_became_active":"2024-04-15T15:21:11.662827+0000","last_became_peered":"2024-04-15T15:21:11.662827+0000","last_unstale":"2024-04-15T15:21:16.691307+0000","last_undegraded":"2024-04-15T15:21:16.691307+0000","last_fullsized":"2024-04-15T15:21:16.691307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:15.476006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730374+0000","last_change":"2024-04-15T15:21:10.636999+0000","last_active":"2024-04-15T15:21:16.730374+0000","last_peered":"2024-04-15T15:21:16.730374+0000","last_clean":"2024-04-15T15:21:16.730374+0000","last_became_active":"2024-04-15T15:21:10.636729+0000","last_became_peered":"2024-04-15T15:21:10.636729+0000","last_unstale":"2024-04-15T15:21:16.730374+0000","last_undegraded":"2024-04-15T15:21:16.730374+0000","last_fullsized":"2024-04-15T15:21:16.730374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730116+0000","last_change":"2024-04-15T15:21:15.708002+0000","last_active":"2024-04-15T15:21:16.730116+0000","last_peered":"2024-04-15T15:21:16.730116+0000","last_clean":"2024-04-15T15:21:16.730116+0000","last_became_active":"2024-04-15T15:21:15.707636+0000","last_became_peered":"2024-04-15T15:21:15.707636+0000","last_unstale":"2024-04-15T15:21:16.730116+0000","last_undegraded":"2024-04-15T15:21:16.730116+0000","last_fullsized":"2024-04-15T15:21:16.730116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730705+0000","last_change":"2024-04-15T15:21:11.662157+0000","last_active":"2024-04-15T15:21:16.730705+0000","last_peered":"2024-04-15T15:21:16.730705+0000","last_clean":"2024-04-15T15:21:16.730705+0000","last_became_active":"2024-04-15T15:21:11.661885+0000","last_became_peered":"2024-04-15T15:21:11.661885+0000","last_unstale":"2024-04-15T15:21:16.730705+0000","last_undegraded":"2024-04-15T15:21:16.730705+0000","last_fullsized":"2024-04-15T15:21:16.730705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:30.561749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731734+0000","last_change":"2024-04-15T15:21:10.635657+0000","last_active":"2024-04-15T15:21:16.731734+0000","last_peered":"2024-04-15T15:21:16.731734+0000","last_clean":"2024-04-15T15:21:16.731734+0000","last_became_active":"2024-04-15T15:21:10.635339+0000","last_became_peered":"2024-04-15T15:21:10.635339+0000","last_unstale":"2024-04-15T15:21:16.731734+0000","last_undegraded":"2024-04-15T15:21:16.731734+0000","last_fullsized":"2024-04-15T15:21:16.731734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690911+0000","last_change":"2024-04-15T15:21:15.902642+0000","last_active":"2024-04-15T15:21:16.690911+0000","last_peered":"2024-04-15T15:21:16.690911+0000","last_clean":"2024-04-15T15:21:16.690911+0000","last_became_active":"2024-04-15T15:21:15.902252+0000","last_became_peered":"2024-04-15T15:21:15.902252+0000","last_unstale":"2024-04-15T15:21:16.690911+0000","last_undegraded":"2024-04-15T15:21:16.690911+0000","last_fullsized":"2024-04-15T15:21:16.690911+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:13.820849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730900+0000","last_change":"2024-04-15T15:21:11.658612+0000","last_active":"2024-04-15T15:21:16.730900+0000","last_peered":"2024-04-15T15:21:16.730900+0000","last_clean":"2024-04-15T15:21:16.730900+0000","last_became_active":"2024-04-15T15:21:11.658312+0000","last_became_peered":"2024-04-15T15:21:11.658312+0000","last_unstale":"2024-04-15T15:21:16.730900+0000","last_undegraded":"2024-04-15T15:21:16.730900+0000","last_fullsized":"2024-04-15T15:21:16.730900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:15.476006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685731+0000","last_change":"2024-04-15T15:21:10.636964+0000","last_active":"2024-04-15T15:21:16.685731+0000","last_peered":"2024-04-15T15:21:16.685731+0000","last_clean":"2024-04-15T15:21:16.685731+0000","last_became_active":"2024-04-15T15:21:10.636641+0000","last_became_peered":"2024-04-15T15:21:10.636641+0000","last_unstale":"2024-04-15T15:21:16.685731+0000","last_undegraded":"2024-04-15T15:21:16.685731+0000","last_fullsized":"2024-04-15T15:21:16.685731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:27:05.012468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.684974+0000","last_change":"2024-04-15T15:21:15.706953+0000","last_active":"2024-04-15T15:21:16.684974+0000","last_peered":"2024-04-15T15:21:16.684974+0000","last_clean":"2024-04-15T15:21:16.684974+0000","last_became_active":"2024-04-15T15:21:15.706465+0000","last_became_peered":"2024-04-15T15:21:15.706465+0000","last_unstale":"2024-04-15T15:21:16.684974+0000","last_undegraded":"2024-04-15T15:21:16.684974+0000","last_fullsized":"2024-04-15T15:21:16.684974+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:06.410056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731056+0000","last_change":"2024-04-15T15:21:11.658373+0000","last_active":"2024-04-15T15:21:16.731056+0000","last_peered":"2024-04-15T15:21:16.731056+0000","last_clean":"2024-04-15T15:21:16.731056+0000","last_became_active":"2024-04-15T15:21:11.658069+0000","last_became_peered":"2024-04-15T15:21:11.658069+0000","last_unstale":"2024-04-15T15:21:16.731056+0000","last_undegraded":"2024-04-15T15:21:16.731056+0000","last_fullsized":"2024-04-15T15:21:16.731056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:59:31.320816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730149+0000","last_change":"2024-04-15T15:21:10.637607+0000","last_active":"2024-04-15T15:21:16.730149+0000","last_peered":"2024-04-15T15:21:16.730149+0000","last_clean":"2024-04-15T15:21:16.730149+0000","last_became_active":"2024-04-15T15:21:10.637277+0000","last_became_peered":"2024-04-15T15:21:10.637277+0000","last_unstale":"2024-04-15T15:21:16.730149+0000","last_undegraded":"2024-04-15T15:21:16.730149+0000","last_fullsized":"2024-04-15T15:21:16.730149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:03:48.192888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731366+0000","last_change":"2024-04-15T15:21:15.696964+0000","last_active":"2024-04-15T15:21:16.731366+0000","last_peered":"2024-04-15T15:21:16.731366+0000","last_clean":"2024-04-15T15:21:16.731366+0000","last_became_active":"2024-04-15T15:21:15.696559+0000","last_became_peered":"2024-04-15T15:21:15.696559+0000","last_unstale":"2024-04-15T15:21:16.731366+0000","last_undegraded":"2024-04-15T15:21:16.731366+0000","last_fullsized":"2024-04-15T15:21:16.731366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:35.384335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729725+0000","last_change":"2024-04-15T15:21:11.661036+0000","last_active":"2024-04-15T15:21:16.729725+0000","last_peered":"2024-04-15T15:21:16.729725+0000","last_clean":"2024-04-15T15:21:16.729725+0000","last_became_active":"2024-04-15T15:21:11.660616+0000","last_became_peered":"2024-04-15T15:21:11.660616+0000","last_unstale":"2024-04-15T15:21:16.729725+0000","last_undegraded":"2024-04-15T15:21:16.729725+0000","last_fullsized":"2024-04-15T15:21:16.729725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729989+0000","last_change":"2024-04-15T15:21:10.644599+0000","last_active":"2024-04-15T15:21:16.729989+0000","last_peered":"2024-04-15T15:21:16.729989+0000","last_clean":"2024-04-15T15:21:16.729989+0000","last_became_active":"2024-04-15T15:21:10.644274+0000","last_became_peered":"2024-04-15T15:21:10.644274+0000","last_unstale":"2024-04-15T15:21:16.729989+0000","last_undegraded":"2024-04-15T15:21:16.729989+0000","last_fullsized":"2024-04-15T15:21:16.729989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:32:53.285490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732952+0000","last_change":"2024-04-15T15:21:15.901481+0000","last_active":"2024-04-15T15:21:16.732952+0000","last_peered":"2024-04-15T15:21:16.732952+0000","last_clean":"2024-04-15T15:21:16.732952+0000","last_became_active":"2024-04-15T15:21:15.901096+0000","last_became_peered":"2024-04-15T15:21:15.901096+0000","last_unstale":"2024-04-15T15:21:16.732952+0000","last_undegraded":"2024-04-15T15:21:16.732952+0000","last_fullsized":"2024-04-15T15:21:16.732952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:38.490295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730312+0000","last_change":"2024-04-15T15:21:11.658341+0000","last_active":"2024-04-15T15:21:16.730312+0000","last_peered":"2024-04-15T15:21:16.730312+0000","last_clean":"2024-04-15T15:21:16.730312+0000","last_became_active":"2024-04-15T15:21:11.658024+0000","last_became_peered":"2024-04-15T15:21:11.658024+0000","last_unstale":"2024-04-15T15:21:16.730312+0000","last_undegraded":"2024-04-15T15:21:16.730312+0000","last_fullsized":"2024-04-15T15:21:16.730312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.568336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691308+0000","last_change":"2024-04-15T15:21:10.645818+0000","last_active":"2024-04-15T15:21:16.691308+0000","last_peered":"2024-04-15T15:21:16.691308+0000","last_clean":"2024-04-15T15:21:16.691308+0000","last_became_active":"2024-04-15T15:21:10.645371+0000","last_became_peered":"2024-04-15T15:21:10.645371+0000","last_unstale":"2024-04-15T15:21:16.691308+0000","last_undegraded":"2024-04-15T15:21:16.691308+0000","last_fullsized":"2024-04-15T15:21:16.691308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733065+0000","last_change":"2024-04-15T15:21:15.899911+0000","last_active":"2024-04-15T15:21:16.733065+0000","last_peered":"2024-04-15T15:21:16.733065+0000","last_clean":"2024-04-15T15:21:16.733065+0000","last_became_active":"2024-04-15T15:21:15.899428+0000","last_became_peered":"2024-04-15T15:21:15.899428+0000","last_unstale":"2024-04-15T15:21:16.733065+0000","last_undegraded":"2024-04-15T15:21:16.733065+0000","last_fullsized":"2024-04-15T15:21:16.733065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:18:40.268133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731100+0000","last_change":"2024-04-15T15:21:11.659032+0000","last_active":"2024-04-15T15:21:16.731100+0000","last_peered":"2024-04-15T15:21:16.731100+0000","last_clean":"2024-04-15T15:21:16.731100+0000","last_became_active":"2024-04-15T15:21:11.658769+0000","last_became_peered":"2024-04-15T15:21:11.658769+0000","last_unstale":"2024-04-15T15:21:16.731100+0000","last_undegraded":"2024-04-15T15:21:16.731100+0000","last_fullsized":"2024-04-15T15:21:16.731100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730358+0000","last_change":"2024-04-15T15:21:10.640715+0000","last_active":"2024-04-15T15:21:16.730358+0000","last_peered":"2024-04-15T15:21:16.730358+0000","last_clean":"2024-04-15T15:21:16.730358+0000","last_became_active":"2024-04-15T15:21:10.640364+0000","last_became_peered":"2024-04-15T15:21:10.640364+0000","last_unstale":"2024-04-15T15:21:16.730358+0000","last_undegraded":"2024-04-15T15:21:16.730358+0000","last_fullsized":"2024-04-15T15:21:16.730358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:54:53.368896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691556+0000","last_change":"2024-04-15T15:21:15.901574+0000","last_active":"2024-04-15T15:21:16.691556+0000","last_peered":"2024-04-15T15:21:16.691556+0000","last_clean":"2024-04-15T15:21:16.691556+0000","last_became_active":"2024-04-15T15:21:15.901144+0000","last_became_peered":"2024-04-15T15:21:15.901144+0000","last_unstale":"2024-04-15T15:21:16.691556+0000","last_undegraded":"2024-04-15T15:21:16.691556+0000","last_fullsized":"2024-04-15T15:21:16.691556+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:11:34.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730711+0000","last_change":"2024-04-15T15:21:11.655252+0000","last_active":"2024-04-15T15:21:16.730711+0000","last_peered":"2024-04-15T15:21:16.730711+0000","last_clean":"2024-04-15T15:21:16.730711+0000","last_became_active":"2024-04-15T15:21:11.654839+0000","last_became_peered":"2024-04-15T15:21:11.654839+0000","last_unstale":"2024-04-15T15:21:16.730711+0000","last_undegraded":"2024-04-15T15:21:16.730711+0000","last_fullsized":"2024-04-15T15:21:16.730711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:08.032185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:21.251862+0000","last_change":"2024-04-15T15:21:10.645952+0000","last_active":"2024-04-15T15:21:21.251862+0000","last_peered":"2024-04-15T15:21:21.251862+0000","last_clean":"2024-04-15T15:21:21.251862+0000","last_became_active":"2024-04-15T15:21:10.645618+0000","last_became_peered":"2024-04-15T15:21:10.645618+0000","last_unstale":"2024-04-15T15:21:21.251862+0000","last_undegraded":"2024-04-15T15:21:21.251862+0000","last_fullsized":"2024-04-15T15:21:21.251862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730096+0000","last_change":"2024-04-15T15:21:15.703360+0000","last_active":"2024-04-15T15:21:16.730096+0000","last_peered":"2024-04-15T15:21:16.730096+0000","last_clean":"2024-04-15T15:21:16.730096+0000","last_became_active":"2024-04-15T15:21:15.702868+0000","last_became_peered":"2024-04-15T15:21:15.702868+0000","last_unstale":"2024-04-15T15:21:16.730096+0000","last_undegraded":"2024-04-15T15:21:16.730096+0000","last_fullsized":"2024-04-15T15:21:16.730096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:19.526528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685743+0000","last_change":"2024-04-15T15:21:11.659799+0000","last_active":"2024-04-15T15:21:16.685743+0000","last_peered":"2024-04-15T15:21:16.685743+0000","last_clean":"2024-04-15T15:21:16.685743+0000","last_became_active":"2024-04-15T15:21:11.659521+0000","last_became_peered":"2024-04-15T15:21:11.659521+0000","last_unstale":"2024-04-15T15:21:16.685743+0000","last_undegraded":"2024-04-15T15:21:16.685743+0000","last_fullsized":"2024-04-15T15:21:16.685743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:57:38.472181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730267+0000","last_change":"2024-04-15T15:21:10.645938+0000","last_active":"2024-04-15T15:21:16.730267+0000","last_peered":"2024-04-15T15:21:16.730267+0000","last_clean":"2024-04-15T15:21:16.730267+0000","last_became_active":"2024-04-15T15:21:10.645642+0000","last_became_peered":"2024-04-15T15:21:10.645642+0000","last_unstale":"2024-04-15T15:21:16.730267+0000","last_undegraded":"2024-04-15T15:21:16.730267+0000","last_fullsized":"2024-04-15T15:21:16.730267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685481+0000","last_change":"2024-04-15T15:21:15.705076+0000","last_active":"2024-04-15T15:21:16.685481+0000","last_peered":"2024-04-15T15:21:16.685481+0000","last_clean":"2024-04-15T15:21:16.685481+0000","last_became_active":"2024-04-15T15:21:15.704627+0000","last_became_peered":"2024-04-15T15:21:15.704627+0000","last_unstale":"2024-04-15T15:21:16.685481+0000","last_undegraded":"2024-04-15T15:21:16.685481+0000","last_fullsized":"2024-04-15T15:21:16.685481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:24.117458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731387+0000","last_change":"2024-04-15T15:21:11.665544+0000","last_active":"2024-04-15T15:21:16.731387+0000","last_peered":"2024-04-15T15:21:16.731387+0000","last_clean":"2024-04-15T15:21:16.731387+0000","last_became_active":"2024-04-15T15:21:11.665263+0000","last_became_peered":"2024-04-15T15:21:11.665263+0000","last_unstale":"2024-04-15T15:21:16.731387+0000","last_undegraded":"2024-04-15T15:21:16.731387+0000","last_fullsized":"2024-04-15T15:21:16.731387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691599+0000","last_change":"2024-04-15T15:21:10.648496+0000","last_active":"2024-04-15T15:21:16.691599+0000","last_peered":"2024-04-15T15:21:16.691599+0000","last_clean":"2024-04-15T15:21:16.691599+0000","last_became_active":"2024-04-15T15:21:10.648222+0000","last_became_peered":"2024-04-15T15:21:10.648222+0000","last_unstale":"2024-04-15T15:21:16.691599+0000","last_undegraded":"2024-04-15T15:21:16.691599+0000","last_fullsized":"2024-04-15T15:21:16.691599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:11.980260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732523+0000","last_change":"2024-04-15T15:21:15.699583+0000","last_active":"2024-04-15T15:21:16.732523+0000","last_peered":"2024-04-15T15:21:16.732523+0000","last_clean":"2024-04-15T15:21:16.732523+0000","last_became_active":"2024-04-15T15:21:15.699119+0000","last_became_peered":"2024-04-15T15:21:15.699119+0000","last_unstale":"2024-04-15T15:21:16.732523+0000","last_undegraded":"2024-04-15T15:21:16.732523+0000","last_fullsized":"2024-04-15T15:21:16.732523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:41:57.140691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691502+0000","last_change":"2024-04-15T15:21:11.663796+0000","last_active":"2024-04-15T15:21:16.691502+0000","last_peered":"2024-04-15T15:21:16.691502+0000","last_clean":"2024-04-15T15:21:16.691502+0000","last_became_active":"2024-04-15T15:21:11.663502+0000","last_became_peered":"2024-04-15T15:21:11.663502+0000","last_unstale":"2024-04-15T15:21:16.691502+0000","last_undegraded":"2024-04-15T15:21:16.691502+0000","last_fullsized":"2024-04-15T15:21:16.691502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:23:31.333813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730207+0000","last_change":"2024-04-15T15:21:10.651176+0000","last_active":"2024-04-15T15:21:16.730207+0000","last_peered":"2024-04-15T15:21:16.730207+0000","last_clean":"2024-04-15T15:21:16.730207+0000","last_became_active":"2024-04-15T15:21:10.650301+0000","last_became_peered":"2024-04-15T15:21:10.650301+0000","last_unstale":"2024-04-15T15:21:16.730207+0000","last_undegraded":"2024-04-15T15:21:16.730207+0000","last_fullsized":"2024-04-15T15:21:16.730207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:52.923241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732313+0000","last_change":"2024-04-15T15:21:15.899209+0000","last_active":"2024-04-15T15:21:16.732313+0000","last_peered":"2024-04-15T15:21:16.732313+0000","last_clean":"2024-04-15T15:21:16.732313+0000","last_became_active":"2024-04-15T15:21:15.898799+0000","last_became_peered":"2024-04-15T15:21:15.898799+0000","last_unstale":"2024-04-15T15:21:16.732313+0000","last_undegraded":"2024-04-15T15:21:16.732313+0000","last_fullsized":"2024-04-15T15:21:16.732313+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:16.350687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731404+0000","last_change":"2024-04-15T15:21:15.902104+0000","last_active":"2024-04-15T15:21:16.731404+0000","last_peered":"2024-04-15T15:21:16.731404+0000","last_clean":"2024-04-15T15:21:16.731404+0000","last_became_active":"2024-04-15T15:21:15.901711+0000","last_became_peered":"2024-04-15T15:21:15.901711+0000","last_unstale":"2024-04-15T15:21:16.731404+0000","last_undegraded":"2024-04-15T15:21:16.731404+0000","last_fullsized":"2024-04-15T15:21:16.731404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:30.138777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729279+0000","last_change":"2024-04-15T15:21:11.649924+0000","last_active":"2024-04-15T15:21:16.729279+0000","last_peered":"2024-04-15T15:21:16.729279+0000","last_clean":"2024-04-15T15:21:16.729279+0000","last_became_active":"2024-04-15T15:21:11.649654+0000","last_became_peered":"2024-04-15T15:21:11.649654+0000","last_unstale":"2024-04-15T15:21:16.729279+0000","last_undegraded":"2024-04-15T15:21:16.729279+0000","last_fullsized":"2024-04-15T15:21:16.729279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691073+0000","last_change":"2024-04-15T15:21:10.647906+0000","last_active":"2024-04-15T15:21:16.691073+0000","last_peered":"2024-04-15T15:21:16.691073+0000","last_clean":"2024-04-15T15:21:16.691073+0000","last_became_active":"2024-04-15T15:21:10.647594+0000","last_became_peered":"2024-04-15T15:21:10.647594+0000","last_unstale":"2024-04-15T15:21:16.691073+0000","last_undegraded":"2024-04-15T15:21:16.691073+0000","last_fullsized":"2024-04-15T15:21:16.691073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729962+0000","last_change":"2024-04-15T15:21:15.707054+0000","last_active":"2024-04-15T15:21:16.729962+0000","last_peered":"2024-04-15T15:21:16.729962+0000","last_clean":"2024-04-15T15:21:16.729962+0000","last_became_active":"2024-04-15T15:21:15.706636+0000","last_became_peered":"2024-04-15T15:21:15.706636+0000","last_unstale":"2024-04-15T15:21:16.729962+0000","last_undegraded":"2024-04-15T15:21:16.729962+0000","last_fullsized":"2024-04-15T15:21:16.729962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:51.618858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732275+0000","last_change":"2024-04-15T15:21:11.644995+0000","last_active":"2024-04-15T15:21:16.732275+0000","last_peered":"2024-04-15T15:21:16.732275+0000","last_clean":"2024-04-15T15:21:16.732275+0000","last_became_active":"2024-04-15T15:21:11.644654+0000","last_became_peered":"2024-04-15T15:21:11.644654+0000","last_unstale":"2024-04-15T15:21:16.732275+0000","last_undegraded":"2024-04-15T15:21:16.732275+0000","last_fullsized":"2024-04-15T15:21:16.732275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732479+0000","last_change":"2024-04-15T15:21:10.645892+0000","last_active":"2024-04-15T15:21:16.732479+0000","last_peered":"2024-04-15T15:21:16.732479+0000","last_clean":"2024-04-15T15:21:16.732479+0000","last_became_active":"2024-04-15T15:21:10.645615+0000","last_became_peered":"2024-04-15T15:21:10.645615+0000","last_unstale":"2024-04-15T15:21:16.732479+0000","last_undegraded":"2024-04-15T15:21:16.732479+0000","last_fullsized":"2024-04-15T15:21:16.732479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:16:25.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730156+0000","last_change":"2024-04-15T15:21:10.646124+0000","last_active":"2024-04-15T15:21:16.730156+0000","last_peered":"2024-04-15T15:21:16.730156+0000","last_clean":"2024-04-15T15:21:16.730156+0000","last_became_active":"2024-04-15T15:21:10.644818+0000","last_became_peered":"2024-04-15T15:21:10.644818+0000","last_unstale":"2024-04-15T15:21:16.730156+0000","last_undegraded":"2024-04-15T15:21:16.730156+0000","last_fullsized":"2024-04-15T15:21:16.730156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:08.844351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730262+0000","last_change":"2024-04-15T15:21:11.648824+0000","last_active":"2024-04-15T15:21:16.730262+0000","last_peered":"2024-04-15T15:21:16.730262+0000","last_clean":"2024-04-15T15:21:16.730262+0000","last_became_active":"2024-04-15T15:21:11.648490+0000","last_became_peered":"2024-04-15T15:21:11.648490+0000","last_unstale":"2024-04-15T15:21:16.730262+0000","last_undegraded":"2024-04-15T15:21:16.730262+0000","last_fullsized":"2024-04-15T15:21:16.730262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730777+0000","last_change":"2024-04-15T15:21:15.699586+0000","last_active":"2024-04-15T15:21:16.730777+0000","last_peered":"2024-04-15T15:21:16.730777+0000","last_clean":"2024-04-15T15:21:16.730777+0000","last_became_active":"2024-04-15T15:21:15.699130+0000","last_became_peered":"2024-04-15T15:21:15.699130+0000","last_unstale":"2024-04-15T15:21:16.730777+0000","last_undegraded":"2024-04-15T15:21:16.730777+0000","last_fullsized":"2024-04-15T15:21:16.730777+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:47.552269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732768+0000","last_change":"2024-04-15T15:21:10.641801+0000","last_active":"2024-04-15T15:21:16.732768+0000","last_peered":"2024-04-15T15:21:16.732768+0000","last_clean":"2024-04-15T15:21:16.732768+0000","last_became_active":"2024-04-15T15:21:10.641511+0000","last_became_peered":"2024-04-15T15:21:10.641511+0000","last_unstale":"2024-04-15T15:21:16.732768+0000","last_undegraded":"2024-04-15T15:21:16.732768+0000","last_fullsized":"2024-04-15T15:21:16.732768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:09.603398+0000","last_clean_scrub_stamp":"2024-04-15T15:21:09.603398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:03:48.192888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685152+0000","last_change":"2024-04-15T15:21:11.650188+0000","last_active":"2024-04-15T15:21:16.685152+0000","last_peered":"2024-04-15T15:21:16.685152+0000","last_clean":"2024-04-15T15:21:16.685152+0000","last_became_active":"2024-04-15T15:21:11.649838+0000","last_became_peered":"2024-04-15T15:21:11.649838+0000","last_unstale":"2024-04-15T15:21:16.685152+0000","last_undegraded":"2024-04-15T15:21:16.685152+0000","last_fullsized":"2024-04-15T15:21:16.685152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:17:12.989671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732888+0000","last_change":"2024-04-15T15:21:15.699639+0000","last_active":"2024-04-15T15:21:16.732888+0000","last_peered":"2024-04-15T15:21:16.732888+0000","last_clean":"2024-04-15T15:21:16.732888+0000","last_became_active":"2024-04-15T15:21:15.699252+0000","last_became_peered":"2024-04-15T15:21:15.699252+0000","last_unstale":"2024-04-15T15:21:16.732888+0000","last_undegraded":"2024-04-15T15:21:16.732888+0000","last_fullsized":"2024-04-15T15:21:16.732888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:14.663331+0000","last_clean_scrub_stamp":"2024-04-15T15:21:14.663331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:02.471663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729618+0000","last_change":"2024-04-15T15:21:11.662366+0000","last_active":"2024-04-15T15:21:16.729618+0000","last_peered":"2024-04-15T15:21:16.729618+0000","last_clean":"2024-04-15T15:21:16.729618+0000","last_became_active":"2024-04-15T15:21:11.662082+0000","last_became_peered":"2024-04-15T15:21:11.662082+0000","last_unstale":"2024-04-15T15:21:16.729618+0000","last_undegraded":"2024-04-15T15:21:16.729618+0000","last_fullsized":"2024-04-15T15:21:16.729618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731145+0000","last_change":"2024-04-15T15:21:11.667867+0000","last_active":"2024-04-15T15:21:16.731145+0000","last_peered":"2024-04-15T15:21:16.731145+0000","last_clean":"2024-04-15T15:21:16.731145+0000","last_became_active":"2024-04-15T15:21:11.667664+0000","last_became_peered":"2024-04-15T15:21:11.667664+0000","last_unstale":"2024-04-15T15:21:16.731145+0000","last_undegraded":"2024-04-15T15:21:16.731145+0000","last_fullsized":"2024-04-15T15:21:16.731145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:31:32.752437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691237+0000","last_change":"2024-04-15T15:21:11.662949+0000","last_active":"2024-04-15T15:21:16.691237+0000","last_peered":"2024-04-15T15:21:16.691237+0000","last_clean":"2024-04-15T15:21:16.691237+0000","last_became_active":"2024-04-15T15:21:11.662014+0000","last_became_peered":"2024-04-15T15:21:11.662014+0000","last_unstale":"2024-04-15T15:21:16.691237+0000","last_undegraded":"2024-04-15T15:21:16.691237+0000","last_fullsized":"2024-04-15T15:21:16.691237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:20.066936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730170+0000","last_change":"2024-04-15T15:21:11.643845+0000","last_active":"2024-04-15T15:21:16.730170+0000","last_peered":"2024-04-15T15:21:16.730170+0000","last_clean":"2024-04-15T15:21:16.730170+0000","last_became_active":"2024-04-15T15:21:11.643459+0000","last_became_peered":"2024-04-15T15:21:11.643459+0000","last_unstale":"2024-04-15T15:21:16.730170+0000","last_undegraded":"2024-04-15T15:21:16.730170+0000","last_fullsized":"2024-04-15T15:21:16.730170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:43.501747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.733169+0000","last_change":"2024-04-15T15:21:11.649864+0000","last_active":"2024-04-15T15:21:16.733169+0000","last_peered":"2024-04-15T15:21:16.733169+0000","last_clean":"2024-04-15T15:21:16.733169+0000","last_became_active":"2024-04-15T15:21:11.649174+0000","last_became_peered":"2024-04-15T15:21:11.649174+0000","last_unstale":"2024-04-15T15:21:16.733169+0000","last_undegraded":"2024-04-15T15:21:16.733169+0000","last_fullsized":"2024-04-15T15:21:16.733169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:55.392737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730146+0000","last_change":"2024-04-15T15:21:11.649917+0000","last_active":"2024-04-15T15:21:16.730146+0000","last_peered":"2024-04-15T15:21:16.730146+0000","last_clean":"2024-04-15T15:21:16.730146+0000","last_became_active":"2024-04-15T15:21:11.649627+0000","last_became_peered":"2024-04-15T15:21:11.649627+0000","last_unstale":"2024-04-15T15:21:16.730146+0000","last_undegraded":"2024-04-15T15:21:16.730146+0000","last_fullsized":"2024-04-15T15:21:16.730146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730543+0000","last_change":"2024-04-15T15:21:11.666666+0000","last_active":"2024-04-15T15:21:16.730543+0000","last_peered":"2024-04-15T15:21:16.730543+0000","last_clean":"2024-04-15T15:21:16.730543+0000","last_became_active":"2024-04-15T15:21:11.666354+0000","last_became_peered":"2024-04-15T15:21:11.666354+0000","last_unstale":"2024-04-15T15:21:16.730543+0000","last_undegraded":"2024-04-15T15:21:16.730543+0000","last_fullsized":"2024-04-15T15:21:16.730543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732965+0000","last_change":"2024-04-15T15:21:11.649875+0000","last_active":"2024-04-15T15:21:16.732965+0000","last_peered":"2024-04-15T15:21:16.732965+0000","last_clean":"2024-04-15T15:21:16.732965+0000","last_became_active":"2024-04-15T15:21:11.649492+0000","last_became_peered":"2024-04-15T15:21:11.649492+0000","last_unstale":"2024-04-15T15:21:16.732965+0000","last_undegraded":"2024-04-15T15:21:16.732965+0000","last_fullsized":"2024-04-15T15:21:16.732965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690980+0000","last_change":"2024-04-15T15:21:11.663545+0000","last_active":"2024-04-15T15:21:16.690980+0000","last_peered":"2024-04-15T15:21:16.690980+0000","last_clean":"2024-04-15T15:21:16.690980+0000","last_became_active":"2024-04-15T15:21:11.662983+0000","last_became_peered":"2024-04-15T15:21:11.662983+0000","last_unstale":"2024-04-15T15:21:16.690980+0000","last_undegraded":"2024-04-15T15:21:16.690980+0000","last_fullsized":"2024-04-15T15:21:16.690980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:24.468426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732374+0000","last_change":"2024-04-15T15:21:11.649581+0000","last_active":"2024-04-15T15:21:16.732374+0000","last_peered":"2024-04-15T15:21:16.732374+0000","last_clean":"2024-04-15T15:21:16.732374+0000","last_became_active":"2024-04-15T15:21:11.649255+0000","last_became_peered":"2024-04-15T15:21:11.649255+0000","last_unstale":"2024-04-15T15:21:16.732374+0000","last_undegraded":"2024-04-15T15:21:16.732374+0000","last_fullsized":"2024-04-15T15:21:16.732374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691111+0000","last_change":"2024-04-15T15:21:11.667227+0000","last_active":"2024-04-15T15:21:16.691111+0000","last_peered":"2024-04-15T15:21:16.691111+0000","last_clean":"2024-04-15T15:21:16.691111+0000","last_became_active":"2024-04-15T15:21:11.663970+0000","last_became_peered":"2024-04-15T15:21:11.663970+0000","last_unstale":"2024-04-15T15:21:16.691111+0000","last_undegraded":"2024-04-15T15:21:16.691111+0000","last_fullsized":"2024-04-15T15:21:16.691111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:02.359534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729911+0000","last_change":"2024-04-15T15:21:11.659069+0000","last_active":"2024-04-15T15:21:16.729911+0000","last_peered":"2024-04-15T15:21:16.729911+0000","last_clean":"2024-04-15T15:21:16.729911+0000","last_became_active":"2024-04-15T15:21:11.658826+0000","last_became_peered":"2024-04-15T15:21:11.658826+0000","last_unstale":"2024-04-15T15:21:16.729911+0000","last_undegraded":"2024-04-15T15:21:16.729911+0000","last_fullsized":"2024-04-15T15:21:16.729911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:34:00.069973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730161+0000","last_change":"2024-04-15T15:21:11.662189+0000","last_active":"2024-04-15T15:21:16.730161+0000","last_peered":"2024-04-15T15:21:16.730161+0000","last_clean":"2024-04-15T15:21:16.730161+0000","last_became_active":"2024-04-15T15:21:11.661850+0000","last_became_peered":"2024-04-15T15:21:11.661850+0000","last_unstale":"2024-04-15T15:21:16.730161+0000","last_undegraded":"2024-04-15T15:21:16.730161+0000","last_fullsized":"2024-04-15T15:21:16.730161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730978+0000","last_change":"2024-04-15T15:21:11.656094+0000","last_active":"2024-04-15T15:21:16.730978+0000","last_peered":"2024-04-15T15:21:16.730978+0000","last_clean":"2024-04-15T15:21:16.730978+0000","last_became_active":"2024-04-15T15:21:11.655585+0000","last_became_peered":"2024-04-15T15:21:11.655585+0000","last_unstale":"2024-04-15T15:21:16.730978+0000","last_undegraded":"2024-04-15T15:21:16.730978+0000","last_fullsized":"2024-04-15T15:21:16.730978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:26.088255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731645+0000","last_change":"2024-04-15T15:21:11.663357+0000","last_active":"2024-04-15T15:21:16.731645+0000","last_peered":"2024-04-15T15:21:16.731645+0000","last_clean":"2024-04-15T15:21:16.731645+0000","last_became_active":"2024-04-15T15:21:11.663051+0000","last_became_peered":"2024-04-15T15:21:11.663051+0000","last_unstale":"2024-04-15T15:21:16.731645+0000","last_undegraded":"2024-04-15T15:21:16.731645+0000","last_fullsized":"2024-04-15T15:21:16.731645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:53.740250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691911+0000","last_change":"2024-04-15T15:21:11.661796+0000","last_active":"2024-04-15T15:21:16.691911+0000","last_peered":"2024-04-15T15:21:16.691911+0000","last_clean":"2024-04-15T15:21:16.691911+0000","last_became_active":"2024-04-15T15:21:11.661288+0000","last_became_peered":"2024-04-15T15:21:11.661288+0000","last_unstale":"2024-04-15T15:21:16.691911+0000","last_undegraded":"2024-04-15T15:21:16.691911+0000","last_fullsized":"2024-04-15T15:21:16.691911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:26.088255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690954+0000","last_change":"2024-04-15T15:21:11.663287+0000","last_active":"2024-04-15T15:21:16.690954+0000","last_peered":"2024-04-15T15:21:16.690954+0000","last_clean":"2024-04-15T15:21:16.690954+0000","last_became_active":"2024-04-15T15:21:11.662684+0000","last_became_peered":"2024-04-15T15:21:11.662684+0000","last_unstale":"2024-04-15T15:21:16.690954+0000","last_undegraded":"2024-04-15T15:21:16.690954+0000","last_fullsized":"2024-04-15T15:21:16.690954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.729440+0000","last_change":"2024-04-15T15:21:11.657278+0000","last_active":"2024-04-15T15:21:16.729440+0000","last_peered":"2024-04-15T15:21:16.729440+0000","last_clean":"2024-04-15T15:21:16.729440+0000","last_became_active":"2024-04-15T15:21:11.656878+0000","last_became_peered":"2024-04-15T15:21:11.656878+0000","last_unstale":"2024-04-15T15:21:16.729440+0000","last_undegraded":"2024-04-15T15:21:16.729440+0000","last_fullsized":"2024-04-15T15:21:16.729440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:18.989044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730993+0000","last_change":"2024-04-15T15:21:11.646900+0000","last_active":"2024-04-15T15:21:16.730993+0000","last_peered":"2024-04-15T15:21:16.730993+0000","last_clean":"2024-04-15T15:21:16.730993+0000","last_became_active":"2024-04-15T15:21:11.646504+0000","last_became_peered":"2024-04-15T15:21:11.646504+0000","last_unstale":"2024-04-15T15:21:16.730993+0000","last_undegraded":"2024-04-15T15:21:16.730993+0000","last_fullsized":"2024-04-15T15:21:16.730993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:01:13.521997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732733+0000","last_change":"2024-04-15T15:21:11.666961+0000","last_active":"2024-04-15T15:21:16.732733+0000","last_peered":"2024-04-15T15:21:16.732733+0000","last_clean":"2024-04-15T15:21:16.732733+0000","last_became_active":"2024-04-15T15:21:11.666715+0000","last_became_peered":"2024-04-15T15:21:11.666715+0000","last_unstale":"2024-04-15T15:21:16.732733+0000","last_undegraded":"2024-04-15T15:21:16.732733+0000","last_fullsized":"2024-04-15T15:21:16.732733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:59:38.154267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.685618+0000","last_change":"2024-04-15T15:21:11.662851+0000","last_active":"2024-04-15T15:21:16.685618+0000","last_peered":"2024-04-15T15:21:16.685618+0000","last_clean":"2024-04-15T15:21:16.685618+0000","last_became_active":"2024-04-15T15:21:11.662536+0000","last_became_peered":"2024-04-15T15:21:11.662536+0000","last_unstale":"2024-04-15T15:21:16.685618+0000","last_undegraded":"2024-04-15T15:21:16.685618+0000","last_fullsized":"2024-04-15T15:21:16.685618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:34:53.932651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730237+0000","last_change":"2024-04-15T15:21:11.670651+0000","last_active":"2024-04-15T15:21:16.730237+0000","last_peered":"2024-04-15T15:21:16.730237+0000","last_clean":"2024-04-15T15:21:16.730237+0000","last_became_active":"2024-04-15T15:21:11.670354+0000","last_became_peered":"2024-04-15T15:21:11.670354+0000","last_unstale":"2024-04-15T15:21:16.730237+0000","last_undegraded":"2024-04-15T15:21:16.730237+0000","last_fullsized":"2024-04-15T15:21:16.730237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730494+0000","last_change":"2024-04-15T15:21:11.670652+0000","last_active":"2024-04-15T15:21:16.730494+0000","last_peered":"2024-04-15T15:21:16.730494+0000","last_clean":"2024-04-15T15:21:16.730494+0000","last_became_active":"2024-04-15T15:21:11.670396+0000","last_became_peered":"2024-04-15T15:21:11.670396+0000","last_unstale":"2024-04-15T15:21:16.730494+0000","last_undegraded":"2024-04-15T15:21:16.730494+0000","last_fullsized":"2024-04-15T15:21:16.730494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730636+0000","last_change":"2024-04-15T15:21:11.665992+0000","last_active":"2024-04-15T15:21:16.730636+0000","last_peered":"2024-04-15T15:21:16.730636+0000","last_clean":"2024-04-15T15:21:16.730636+0000","last_became_active":"2024-04-15T15:21:11.665531+0000","last_became_peered":"2024-04-15T15:21:11.665531+0000","last_unstale":"2024-04-15T15:21:16.730636+0000","last_undegraded":"2024-04-15T15:21:16.730636+0000","last_fullsized":"2024-04-15T15:21:16.730636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:28:09.770327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.732177+0000","last_change":"2024-04-15T15:21:11.665238+0000","last_active":"2024-04-15T15:21:16.732177+0000","last_peered":"2024-04-15T15:21:16.732177+0000","last_clean":"2024-04-15T15:21:16.732177+0000","last_became_active":"2024-04-15T15:21:11.664931+0000","last_became_peered":"2024-04-15T15:21:11.664931+0000","last_unstale":"2024-04-15T15:21:16.732177+0000","last_undegraded":"2024-04-15T15:21:16.732177+0000","last_fullsized":"2024-04-15T15:21:16.732177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:59:47.664005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731045+0000","last_change":"2024-04-15T15:21:11.655222+0000","last_active":"2024-04-15T15:21:16.731045+0000","last_peered":"2024-04-15T15:21:16.731045+0000","last_clean":"2024-04-15T15:21:16.731045+0000","last_became_active":"2024-04-15T15:21:11.654826+0000","last_became_peered":"2024-04-15T15:21:11.654826+0000","last_unstale":"2024-04-15T15:21:16.731045+0000","last_undegraded":"2024-04-15T15:21:16.731045+0000","last_fullsized":"2024-04-15T15:21:16.731045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:58:20.066936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.690947+0000","last_change":"2024-04-15T15:21:11.662632+0000","last_active":"2024-04-15T15:21:16.690947+0000","last_peered":"2024-04-15T15:21:16.690947+0000","last_clean":"2024-04-15T15:21:16.690947+0000","last_became_active":"2024-04-15T15:21:11.661632+0000","last_became_peered":"2024-04-15T15:21:11.661632+0000","last_unstale":"2024-04-15T15:21:16.690947+0000","last_undegraded":"2024-04-15T15:21:16.690947+0000","last_fullsized":"2024-04-15T15:21:16.690947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:18:22.878184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730098+0000","last_change":"2024-04-15T15:21:11.655451+0000","last_active":"2024-04-15T15:21:16.730098+0000","last_peered":"2024-04-15T15:21:16.730098+0000","last_clean":"2024-04-15T15:21:16.730098+0000","last_became_active":"2024-04-15T15:21:11.655138+0000","last_became_peered":"2024-04-15T15:21:11.655138+0000","last_unstale":"2024-04-15T15:21:16.730098+0000","last_undegraded":"2024-04-15T15:21:16.730098+0000","last_fullsized":"2024-04-15T15:21:16.730098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:28.276724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.691160+0000","last_change":"2024-04-15T15:21:11.662669+0000","last_active":"2024-04-15T15:21:16.691160+0000","last_peered":"2024-04-15T15:21:16.691160+0000","last_clean":"2024-04-15T15:21:16.691160+0000","last_became_active":"2024-04-15T15:21:11.662385+0000","last_became_peered":"2024-04-15T15:21:11.662385+0000","last_unstale":"2024-04-15T15:21:16.691160+0000","last_undegraded":"2024-04-15T15:21:16.691160+0000","last_fullsized":"2024-04-15T15:21:16.691160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:55.392737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.731209+0000","last_change":"2024-04-15T15:21:11.640925+0000","last_active":"2024-04-15T15:21:16.731209+0000","last_peered":"2024-04-15T15:21:16.731209+0000","last_clean":"2024-04-15T15:21:16.731209+0000","last_became_active":"2024-04-15T15:21:11.640618+0000","last_became_peered":"2024-04-15T15:21:11.640618+0000","last_unstale":"2024-04-15T15:21:16.731209+0000","last_undegraded":"2024-04-15T15:21:16.731209+0000","last_fullsized":"2024-04-15T15:21:16.731209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:12:11.318612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730901+0000","last_change":"2024-04-15T15:21:11.653243+0000","last_active":"2024-04-15T15:21:16.730901+0000","last_peered":"2024-04-15T15:21:16.730901+0000","last_clean":"2024-04-15T15:21:16.730901+0000","last_became_active":"2024-04-15T15:21:11.652962+0000","last_became_peered":"2024-04-15T15:21:11.652962+0000","last_unstale":"2024-04-15T15:21:16.730901+0000","last_undegraded":"2024-04-15T15:21:16.730901+0000","last_fullsized":"2024-04-15T15:21:16.730901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:24:50.673642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:21:16.730496+0000","last_change":"2024-04-15T15:21:11.662774+0000","last_active":"2024-04-15T15:21:16.730496+0000","last_peered":"2024-04-15T15:21:16.730496+0000","last_clean":"2024-04-15T15:21:16.730496+0000","last_became_active":"2024-04-15T15:21:11.662488+0000","last_became_peered":"2024-04-15T15:21:11.662488+0000","last_unstale":"2024-04-15T15:21:16.730496+0000","last_undegraded":"2024-04-15T15:21:16.730496+0000","last_fullsized":"2024-04-15T15:21:16.730496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:21:10.612809+0000","last_clean_scrub_stamp":"2024-04-15T15:21:10.612809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:06:09.923283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:21:37.948 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-15T15:21:37.949 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T15:21:37.949 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-15T15:21:37.949 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-15T15:21:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.086+0000 7fd172850640 1 Processor -- start 2024-04-15T15:21:38.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.086+0000 7fd172850640 1 -- start start 2024-04-15T15:21:38.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.086+0000 7fd172850640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd16c0b35f0 0x7fd16c0d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.086+0000 7fd172850640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 0x7fd16c0d1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.086+0000 7fd172850640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd16c0d5190 con 0x7fd16c161fe0 2024-04-15T15:21:38.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.086+0000 7fd172850640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd16c0d19d0 con 0x7fd16c0b35f0 2024-04-15T15:21:38.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.086+0000 7fd172850640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd16c0cf3c0 con 0x7fd16c0d1060 2024-04-15T15:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.087+0000 7fd16b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd16c0b35f0 0x7fd16c0d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.087+0000 7fd16b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd16c0b35f0 0x7fd16c0d3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46952/0 (socket says 172.21.15.12:46952) 2024-04-15T15:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.087+0000 7fd16b7fe640 1 -- 172.21.15.12:0/1262352339 learned_addr learned my addr 172.21.15.12:0/1262352339 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.087+0000 7fd170ded640 1 --2- 172.21.15.12:0/1262352339 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 0x7fd16c0d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.087+0000 7fd16b7fe640 1 -- 172.21.15.12:0/1262352339 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 msgr2=0x7fd16c0d1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.087+0000 7fd16b7fe640 1 --2- 172.21.15.12:0/1262352339 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 0x7fd16c0d1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.087+0000 7fd16b7fe640 1 -- 172.21.15.12:0/1262352339 >> v1:172.21.15.55:6789/0 conn(0x7fd16c161fe0 legacy=0x7fd16c0b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.087+0000 7fd16b7fe640 1 -- 172.21.15.12:0/1262352339 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c0cf6a0 con 0x7fd16c0b35f0 2024-04-15T15:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd16b7fe640 1 --2- 172.21.15.12:0/1262352339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c0d3320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd160009870 tx=0x7fd16002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb94fd7a16e0226 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd16affd640 1 -- 172.21.15.12:0/1262352339 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160034070 con 0x7fd16c0b35f0 2024-04-15T15:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd16affd640 1 -- 172.21.15.12:0/1262352339 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd160030840 con 0x7fd16c0b35f0 2024-04-15T15:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd16affd640 1 -- 172.21.15.12:0/1262352339 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160030c10 con 0x7fd16c0b35f0 2024-04-15T15:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd172850640 1 -- 172.21.15.12:0/1262352339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 msgr2=0x7fd16c0d3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd172850640 1 --2- 172.21.15.12:0/1262352339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c0d3320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd160009870 tx=0x7fd16002b980 comp rx=0 tx=0).stop 2024-04-15T15:21:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd172850640 1 -- 172.21.15.12:0/1262352339 shutdown_connections 2024-04-15T15:21:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd172850640 1 --2- 172.21.15.12:0/1262352339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c0d3320 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd172850640 1 --2- 172.21.15.12:0/1262352339 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 0x7fd16c0d1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd172850640 1 -- 172.21.15.12:0/1262352339 >> 172.21.15.12:0/1262352339 conn(0x7fd16c08bf60 msgr2=0x7fd16c0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd172850640 1 -- 172.21.15.12:0/1262352339 shutdown_connections 2024-04-15T15:21:38.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.088+0000 7fd172850640 1 -- 172.21.15.12:0/1262352339 wait complete. 2024-04-15T15:21:38.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.090+0000 7fd172850640 1 Processor -- start 2024-04-15T15:21:38.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.090+0000 7fd172850640 1 -- start start 2024-04-15T15:21:38.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd172850640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c14f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd172850640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 0x7fd16c0c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd172850640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd16c161fe0 0x7fd16c15cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd172850640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c0d5470 con 0x7fd16c0b35f0 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd172850640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c0b46c0 con 0x7fd16c161fe0 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd172850640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd16c1623c0 con 0x7fd16c0d1060 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c14f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c14f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46968/0 (socket says 172.21.15.12:46968) 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 -- 172.21.15.12:0/2791979736 learned_addr learned my addr 172.21.15.12:0/2791979736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd170ded640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd16c161fe0 0x7fd16c15cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16b7fe640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 0x7fd16c0c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 -- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 msgr2=0x7fd16c0c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 0x7fd16c0c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 -- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd16c161fe0 msgr2=0x7fd16c15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd16c161fe0 0x7fd16c15cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 -- 172.21.15.12:0/2791979736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c0c8eb0 con 0x7fd16c0b35f0 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd16bfff640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c14f970 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c340 tx=0x7fd15c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd168ff9640 1 -- 172.21.15.12:0/2791979736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c017070 con 0x7fd16c0b35f0 2024-04-15T15:21:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd168ff9640 1 -- 172.21.15.12:0/2791979736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15c0052d0 con 0x7fd16c0b35f0 2024-04-15T15:21:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c0c9190 con 0x7fd16c0b35f0 2024-04-15T15:21:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd168ff9640 1 -- 172.21.15.12:0/2791979736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c003850 con 0x7fd16c0b35f0 2024-04-15T15:21:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.091+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c0c9750 con 0x7fd16c0b35f0 2024-04-15T15:21:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.092+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c0c9b90 con 0x7fd16c0b35f0 2024-04-15T15:21:38.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.093+0000 7fd168ff9640 1 -- 172.21.15.12:0/2791979736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd15c01a070 con 0x7fd16c0b35f0 2024-04-15T15:21:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.093+0000 7fd168ff9640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd15806c8f0 0x7fd15806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.093+0000 7fd168ff9640 1 -- 172.21.15.12:0/2791979736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd15c08b210 con 0x7fd16c0b35f0 2024-04-15T15:21:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.094+0000 7fd16b7fe640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd15806c8f0 0x7fd15806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.094+0000 7fd16b7fe640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd15806c8f0 0x7fd15806edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd160036000 tx=0x7fd160008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.098+0000 7fd168ff9640 1 -- 172.21.15.12:0/2791979736 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd15c058b80 con 0x7fd16c0b35f0 2024-04-15T15:21:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.320+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd16c0d0e70 con 0x7fd16c0b35f0 2024-04-15T15:21:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.320+0000 7fd168ff9640 1 -- 172.21.15.12:0/2791979736 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd15c05ce80 con 0x7fd16c0b35f0 2024-04-15T15:21:38.322 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:38.322 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T15:21:38.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd15806c8f0 msgr2=0x7fd15806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd15806c8f0 0x7fd15806edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd160036000 tx=0x7fd160008030 comp rx=0 tx=0).stop 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 msgr2=0x7fd16c14f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c14f970 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c340 tx=0x7fd15c00c800 comp rx=0 tx=0).stop 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 shutdown_connections 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd15806c8f0 0x7fd15806edb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd16c161fe0 0x7fd16c15cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd16c0d1060 0x7fd16c0c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 --2- 172.21.15.12:0/2791979736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd16c0b35f0 0x7fd16c14f970 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 >> 172.21.15.12:0/2791979736 conn(0x7fd16c08bf60 msgr2=0x7fd16c0b5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 shutdown_connections 2024-04-15T15:21:38.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.322+0000 7fd172850640 1 -- 172.21.15.12:0/2791979736 wait complete. 2024-04-15T15:21:38.334 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-15T15:21:38.335 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:38.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.437+0000 7f5aea293640 1 Processor -- start 2024-04-15T15:21:38.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.438+0000 7f5aea293640 1 -- start start 2024-04-15T15:21:38.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.438+0000 7f5aea293640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5ae40701f0 0x7f5ae4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.438+0000 7f5aea293640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae415fb30 0x7f5ae40d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.438+0000 7f5aea293640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae4070720 con 0x7f5ae409a9d0 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.438+0000 7f5aea293640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae409ba90 con 0x7f5ae40701f0 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.438+0000 7f5aea293640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae40d08c0 con 0x7f5ae415fb30 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5ae8830640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae415fb30 0x7f5ae40d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5ae8830640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae415fb30 0x7f5ae40d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:33976/0 (socket says 172.21.15.12:33976) 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5adbfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5ae40701f0 0x7f5ae4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5ae3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f5ae409a9d0 0x7f5ae409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:58890/0 (socket says 172.21.15.12:58890) 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5ae8830640 1 -- 172.21.15.12:0/2691320335 learned_addr learned my addr 172.21.15.12:0/2691320335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5ae8830640 1 -- 172.21.15.12:0/2691320335 >> v2:172.21.15.55:3300/0 conn(0x7f5ae40701f0 msgr2=0x7f5ae4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5ae8830640 1 --2- 172.21.15.12:0/2691320335 >> v2:172.21.15.55:3300/0 conn(0x7f5ae40701f0 0x7f5ae4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5ae8830640 1 -- 172.21.15.12:0/2691320335 >> v1:172.21.15.55:6789/0 conn(0x7f5ae409a9d0 legacy=0x7f5ae409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5ae8830640 1 -- 172.21.15.12:0/2691320335 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae4070a90 con 0x7f5ae415fb30 2024-04-15T15:21:38.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.439+0000 7f5adbfff640 1 --2- 172.21.15.12:0/2691320335 >> v2:172.21.15.55:3300/0 conn(0x7f5ae40701f0 0x7f5ae4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:21:38.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.440+0000 7f5ae8830640 1 --2- 172.21.15.12:0/2691320335 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae415fb30 0x7f5ae40d0d50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009e50 tx=0x7f5adc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb68d99053a6a2b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.440+0000 7f5ae37fe640 1 -- 172.21.15.12:0/2691320335 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc02cb00 con 0x7f5ae415fb30 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.440+0000 7f5ae37fe640 1 -- 172.21.15.12:0/2691320335 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5adc02cca0 con 0x7f5ae415fb30 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.440+0000 7f5ae37fe640 1 -- 172.21.15.12:0/2691320335 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc034610 con 0x7f5ae415fb30 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.441+0000 7f5aea293640 1 -- 172.21.15.12:0/2691320335 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae415fb30 msgr2=0x7f5ae40d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.441+0000 7f5aea293640 1 --2- 172.21.15.12:0/2691320335 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae415fb30 0x7f5ae40d0d50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009e50 tx=0x7f5adc02bf60 comp rx=0 tx=0).stop 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.441+0000 7f5aea293640 1 -- 172.21.15.12:0/2691320335 shutdown_connections 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.441+0000 7f5aea293640 1 --2- 172.21.15.12:0/2691320335 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae415fb30 0x7f5ae40d0d50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.441+0000 7f5aea293640 1 --2- 172.21.15.12:0/2691320335 >> v2:172.21.15.55:3300/0 conn(0x7f5ae40701f0 0x7f5ae4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.441+0000 7f5aea293640 1 -- 172.21.15.12:0/2691320335 >> 172.21.15.12:0/2691320335 conn(0x7f5ae40d2460 msgr2=0x7f5ae40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.441+0000 7f5aea293640 1 -- 172.21.15.12:0/2691320335 shutdown_connections 2024-04-15T15:21:38.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.441+0000 7f5aea293640 1 -- 172.21.15.12:0/2691320335 wait complete. 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.442+0000 7f5aea293640 1 Processor -- start 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.442+0000 7f5aea293640 1 -- start start 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5aea293640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae40701f0 0x7f5ae40b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5aea293640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae409a9d0 0x7f5ae41f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5aea293640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae415fb30 0x7f5ae4154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5aea293640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae415ffc0 con 0x7f5ae409a9d0 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5aea293640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae4071050 con 0x7f5ae415fb30 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5aea293640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae40b1c60 con 0x7f5ae40701f0 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae409a9d0 0x7f5ae41f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae409a9d0 0x7f5ae41f5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46982/0 (socket says 172.21.15.12:46982) 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 -- 172.21.15.12:0/2729112608 learned_addr learned my addr 172.21.15.12:0/2729112608 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:38.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5ae3fff640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae40701f0 0x7f5ae40b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5ae8830640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae415fb30 0x7f5ae4154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 -- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae40701f0 msgr2=0x7f5ae40b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae40701f0 0x7f5ae40b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 -- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae415fb30 msgr2=0x7f5ae4154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae415fb30 0x7f5ae4154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 -- 172.21.15.12:0/2729112608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae40af800 con 0x7f5ae409a9d0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5adbfff640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae409a9d0 0x7f5ae41f5960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00e900 tx=0x7f5acc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.443+0000 7f5ae17fa640 1 -- 172.21.15.12:0/2729112608 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc017070 con 0x7f5ae409a9d0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.444+0000 7f5ae17fa640 1 -- 172.21.15.12:0/2729112608 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5acc00ca00 con 0x7f5ae409a9d0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.444+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae40b1030 con 0x7f5ae409a9d0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.444+0000 7f5ae17fa640 1 -- 172.21.15.12:0/2729112608 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00cdf0 con 0x7f5ae409a9d0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.444+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae40afd80 con 0x7f5ae409a9d0 2024-04-15T15:21:38.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.445+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae41601d0 con 0x7f5ae409a9d0 2024-04-15T15:21:38.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.445+0000 7f5ae17fa640 1 -- 172.21.15.12:0/2729112608 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5acc01a070 con 0x7f5ae409a9d0 2024-04-15T15:21:38.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.445+0000 7f5ae17fa640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ac406c8f0 0x7f5ac406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.445+0000 7f5ae17fa640 1 -- 172.21.15.12:0/2729112608 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5acc08b990 con 0x7f5ae409a9d0 2024-04-15T15:21:38.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.446+0000 7f5ae3fff640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ac406c8f0 0x7f5ac406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.446+0000 7f5ae3fff640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ac406c8f0 0x7f5ac406edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4007fd0 tx=0x7f5ad4007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.450+0000 7f5ae17fa640 1 -- 172.21.15.12:0/2729112608 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ae41601d0 con 0x7f5ae409a9d0 2024-04-15T15:21:38.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.638+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ae4071f70 con 0x7f5ae409a9d0 2024-04-15T15:21:38.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.639+0000 7f5ae17fa640 1 -- 172.21.15.12:0/2729112608 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f5ae4071f70 con 0x7f5ae409a9d0 2024-04-15T15:21:38.641 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:38.641 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:38.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ac406c8f0 msgr2=0x7f5ac406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ac406c8f0 0x7f5ac406edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4007fd0 tx=0x7f5ad4007e80 comp rx=0 tx=0).stop 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae409a9d0 msgr2=0x7f5ae41f5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae409a9d0 0x7f5ae41f5960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00e900 tx=0x7f5acc00edc0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 shutdown_connections 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ac406c8f0 0x7f5ac406edb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ae415fb30 0x7f5ae4154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae409a9d0 0x7f5ae41f5960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 --2- 172.21.15.12:0/2729112608 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5ae40701f0 0x7f5ae40b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 >> 172.21.15.12:0/2729112608 conn(0x7f5ae40d2460 msgr2=0x7f5ae40b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 shutdown_connections 2024-04-15T15:21:38.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.641+0000 7f5aea293640 1 -- 172.21.15.12:0/2729112608 wait complete. 2024-04-15T15:21:38.654 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:38.654 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:38.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.761+0000 7f95600c7640 1 Processor -- start 2024-04-15T15:21:38.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.761+0000 7f95600c7640 1 -- start start 2024-04-15T15:21:38.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.761+0000 7f95600c7640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95580d0690 0x7f95580d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.761+0000 7f95600c7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95580b3160 0x7f955815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.761+0000 7f95600c7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f955815ef70 0x7f955815f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.761+0000 7f95600c7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f95580d0fb0 con 0x7f95580b3160 2024-04-15T15:21:38.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.761+0000 7f95600c7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f955815cb50 con 0x7f955815ef70 2024-04-15T15:21:38.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.761+0000 7f95600c7640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f955815bfd0 con 0x7f95580d0690 2024-04-15T15:21:38.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955d662640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95580b3160 0x7f955815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955d662640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f95580b3160 0x7f955815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46990/0 (socket says 172.21.15.12:46990) 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955d662640 1 -- 172.21.15.12:0/1553870875 learned_addr learned my addr 172.21.15.12:0/1553870875 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955e664640 1 --2- 172.21.15.12:0/1553870875 >> v2:172.21.15.55:3300/0 conn(0x7f955815ef70 0x7f955815f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955de63640 1 --2- 172.21.15.12:0/1553870875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95580d0690 0x7f95580d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955d662640 1 -- 172.21.15.12:0/1553870875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95580d0690 msgr2=0x7f95580d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955d662640 1 --2- 172.21.15.12:0/1553870875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95580d0690 0x7f95580d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955d662640 1 -- 172.21.15.12:0/1553870875 >> v2:172.21.15.55:3300/0 conn(0x7f955815ef70 msgr2=0x7f955815f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955d662640 1 --2- 172.21.15.12:0/1553870875 >> v2:172.21.15.55:3300/0 conn(0x7f955815ef70 0x7f955815f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955d662640 1 -- 172.21.15.12:0/1553870875 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955815c2b0 con 0x7f95580b3160 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955de63640 1 --2- 172.21.15.12:0/1553870875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95580d0690 0x7f95580d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.762+0000 7f955e664640 1 --2- 172.21.15.12:0/1553870875 >> v2:172.21.15.55:3300/0 conn(0x7f955815ef70 0x7f955815f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:38.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f955d662640 1 --2- 172.21.15.12:0/1553870875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f955815b690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f95580b4ba0 tx=0x7f954c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434997e99c03b19d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f955ce61640 1 -- 172.21.15.12:0/1553870875 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c035020 con 0x7f95580b3160 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f955ce61640 1 -- 172.21.15.12:0/1553870875 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f954c02d040 con 0x7f95580b3160 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f955ce61640 1 -- 172.21.15.12:0/1553870875 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c031a80 con 0x7f95580b3160 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f95600c7640 1 -- 172.21.15.12:0/1553870875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 msgr2=0x7f955815b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f95600c7640 1 --2- 172.21.15.12:0/1553870875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f955815b690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f95580b4ba0 tx=0x7f954c02b900 comp rx=0 tx=0).stop 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f95600c7640 1 -- 172.21.15.12:0/1553870875 shutdown_connections 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f95600c7640 1 --2- 172.21.15.12:0/1553870875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f955815b690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f95600c7640 1 --2- 172.21.15.12:0/1553870875 >> v2:172.21.15.55:3300/0 conn(0x7f955815ef70 0x7f955815f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f95600c7640 1 --2- 172.21.15.12:0/1553870875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95580d0690 0x7f95580d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.763+0000 7f95600c7640 1 -- 172.21.15.12:0/1553870875 >> 172.21.15.12:0/1553870875 conn(0x7f955806be00 msgr2=0x7f955806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:38.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.764+0000 7f95600c7640 1 -- 172.21.15.12:0/1553870875 shutdown_connections 2024-04-15T15:21:38.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.764+0000 7f95600c7640 1 -- 172.21.15.12:0/1553870875 wait complete. 2024-04-15T15:21:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f95600c7640 1 Processor -- start 2024-04-15T15:21:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f95600c7640 1 -- start start 2024-04-15T15:21:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f95600c7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f95580cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f95600c7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95580d0690 0x7f95581e6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f95600c7640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f955815ef70 0x7f955806dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f95600c7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f95580b4fb0 con 0x7f95580b3160 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f95600c7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f955815c870 con 0x7f95580d0690 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f95600c7640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f955815bbd0 con 0x7f955815ef70 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.765+0000 7f955de63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f95580cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955de63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f95580cf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46996/0 (socket says 172.21.15.12:46996) 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955de63640 1 -- 172.21.15.12:0/2799128220 learned_addr learned my addr 172.21.15.12:0/2799128220 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955d662640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95580d0690 0x7f95581e6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955de63640 1 -- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f955815ef70 msgr2=0x7f955806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955de63640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f955815ef70 0x7f955806dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955de63640 1 -- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95580d0690 msgr2=0x7f95581e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955de63640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95580d0690 0x7f95581e6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955de63640 1 -- 172.21.15.12:0/2799128220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95580cfa90 con 0x7f95580b3160 2024-04-15T15:21:38.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f955de63640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f95580cf550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f954800ce30 tx=0x7f954800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f95567fc640 1 -- 172.21.15.12:0/2799128220 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954800f680 con 0x7f95580b3160 2024-04-15T15:21:38.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f95567fc640 1 -- 172.21.15.12:0/2799128220 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f954800f820 con 0x7f95580b3160 2024-04-15T15:21:38.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95581e69e0 con 0x7f95580b3160 2024-04-15T15:21:38.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f95567fc640 1 -- 172.21.15.12:0/2799128220 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954800fc10 con 0x7f95580b3160 2024-04-15T15:21:38.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.766+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95581e6fa0 con 0x7f95580b3160 2024-04-15T15:21:38.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.767+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95580689e0 con 0x7f95580b3160 2024-04-15T15:21:38.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.768+0000 7f95567fc640 1 -- 172.21.15.12:0/2799128220 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9548018400 con 0x7f95580b3160 2024-04-15T15:21:38.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.768+0000 7f95567fc640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f954406c8f0 0x7f954406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:38.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.768+0000 7f95567fc640 1 -- 172.21.15.12:0/2799128220 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f954808ccc0 con 0x7f95580b3160 2024-04-15T15:21:38.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.768+0000 7f955d662640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f954406c8f0 0x7f954406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:38.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.769+0000 7f955d662640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f954406c8f0 0x7f954406edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f954c036000 tx=0x7f954c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:38.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.772+0000 7f95567fc640 1 -- 172.21.15.12:0/2799128220 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9548014030 con 0x7f95580b3160 2024-04-15T15:21:38.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.956+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9558073d10 con 0x7f95580b3160 2024-04-15T15:21:38.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.956+0000 7f95567fc640 1 -- 172.21.15.12:0/2799128220 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f954805ad20 con 0x7f95580b3160 2024-04-15T15:21:38.958 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:38.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f954406c8f0 msgr2=0x7f954406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f954406c8f0 0x7f954406edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f954c036000 tx=0x7f954c008030 comp rx=0 tx=0).stop 2024-04-15T15:21:38.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 msgr2=0x7f95580cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:38.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f95580cf550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f954800ce30 tx=0x7f954800a650 comp rx=0 tx=0).stop 2024-04-15T15:21:38.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 shutdown_connections 2024-04-15T15:21:38.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f955815ef70 0x7f955806dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f954406c8f0 0x7f954406edb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95580d0690 0x7f95581e6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 --2- 172.21.15.12:0/2799128220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95580b3160 0x7f95580cf550 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:38.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 >> 172.21.15.12:0/2799128220 conn(0x7f955806be00 msgr2=0x7f9558074df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:38.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 shutdown_connections 2024-04-15T15:21:38.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:38.959+0000 7f95600c7640 1 -- 172.21.15.12:0/2799128220 wait complete. 2024-04-15T15:21:38.971 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.077+0000 7f92a62ce640 1 Processor -- start 2024-04-15T15:21:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.077+0000 7f92a62ce640 1 -- start start 2024-04-15T15:21:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f92a62ce640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 0x7f92a00b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f92a62ce640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a0067e30 con 0x7f92a015e830 2024-04-15T15:21:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f92a62ce640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a00b3c10 con 0x7f92a015ec10 2024-04-15T15:21:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f92a62ce640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f92a00b3ef0 con 0x7f92a00b5ad0 2024-04-15T15:21:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f929ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f92a015e830 0x7f92a0067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56418/0 (socket says 172.21.15.12:56418) 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f929ffff640 1 -- 172.21.15.12:0/2984318233 learned_addr learned my addr 172.21.15.12:0/2984318233 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f92a486b640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 0x7f92a00b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020754820 0 0) 0x7f92a0067e30 con 0x7f92a015e830 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.078+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9288003520 con 0x7f92a015e830 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2966264155 0 0) 0x7f9288003520 con 0x7f92a015e830 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 msgr2=0x7f92a00b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 0x7f92a00b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 >> v1:172.21.15.55:6789/0 conn(0x7f92a015ec10 legacy=0x7f92a00b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a00b41d0 con 0x7f92a015e830 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f9290001670 con 0x7f92a015e830 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f92a486b640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 0x7f92a00b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 >> v1:172.21.15.12:6789/0 conn(0x7f92a015e830 legacy=0x7f92a0067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f9288003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 0x7f9288004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9288004d40 0x7f92880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f92a00b3ef0 con 0x7f92880039e0 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f928800bb50 con 0x7f9288004d40 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f928800bd50 con 0x7f92a00b5ad0 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929ffff640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f9288003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929f7fe640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 0x7f9288004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f92a486b640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9288004d40 0x7f92880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929ffff640 1 -- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 msgr2=0x7f9288004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929ffff640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a00b5ad0 0x7f9288004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929ffff640 1 -- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9288004d40 msgr2=0x7f92880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929ffff640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9288004d40 0x7f92880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f929ffff640 1 -- 172.21.15.12:0/2984318233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9288003520 con 0x7f92880039e0 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.079+0000 7f92a486b640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9288004d40 0x7f92880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.080+0000 7f929ffff640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f9288003e00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f929000c240 tx=0x7f9288005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0e3df72fd4f618 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.080+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290007cf0 con 0x7f92880039e0 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.080+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9290030070 con 0x7f92880039e0 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.080+0000 7f929effd640 1 -- 172.21.15.12:0/2984318233 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290005d30 con 0x7f92880039e0 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.081+0000 7f92a62ce640 1 -- 172.21.15.12:0/2984318233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 msgr2=0x7f9288003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.081+0000 7f92a62ce640 1 --2- 172.21.15.12:0/2984318233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f9288003e00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f929000c240 tx=0x7f9288005830 comp rx=0 tx=0).stop 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.081+0000 7f92a486b640 1 -- 172.21.15.12:0/2984318233 reap_dead start 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.081+0000 7f92a62ce640 1 -- 172.21.15.12:0/2984318233 shutdown_connections 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.081+0000 7f92a62ce640 1 -- 172.21.15.12:0/2984318233 >> 172.21.15.12:0/2984318233 conn(0x7f92a0073360 msgr2=0x7f92a006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.081+0000 7f92a62ce640 1 -- 172.21.15.12:0/2984318233 shutdown_connections 2024-04-15T15:21:39.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.081+0000 7f92a62ce640 1 -- 172.21.15.12:0/2984318233 wait complete. 2024-04-15T15:21:39.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.082+0000 7f92a62ce640 1 Processor -- start 2024-04-15T15:21:39.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.082+0000 7f92a62ce640 1 -- start start 2024-04-15T15:21:39.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f92a62ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f92a01560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f92a62ce640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a0142650 0x7f92a01580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f92a62ce640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a01ea4b0 0x7f92a01ea930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f92a62ce640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f92a015ff60 con 0x7f92880039e0 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f92a62ce640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f92a00b4790 con 0x7f92a01ea4b0 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f92a62ce640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f92a01600e0 con 0x7f92a0142650 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f92a01560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f92a01560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47020/0 (socket says 172.21.15.12:47020) 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 -- 172.21.15.12:0/3054632332 learned_addr learned my addr 172.21.15.12:0/3054632332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929f7fe640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a0142650 0x7f92a01580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f92a486b640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a01ea4b0 0x7f92a01ea930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 -- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a0142650 msgr2=0x7f92a01580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a0142650 0x7f92a01580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 -- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a01ea4b0 msgr2=0x7f92a01ea930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a01ea4b0 0x7f92a01ea930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 -- 172.21.15.12:0/3054632332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a0155040 con 0x7f92880039e0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929ffff640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f92a01560e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f929000c330 tx=0x7f92a00b43a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.083+0000 7f929cff9640 1 -- 172.21.15.12:0/3054632332 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290045070 con 0x7f92880039e0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.084+0000 7f929cff9640 1 -- 172.21.15.12:0/3054632332 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9290000ed0 con 0x7f92880039e0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.084+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a0155320 con 0x7f92880039e0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.084+0000 7f929cff9640 1 -- 172.21.15.12:0/3054632332 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290001c30 con 0x7f92880039e0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.084+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a01558e0 con 0x7f92880039e0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.085+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a01eae70 con 0x7f92880039e0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.085+0000 7f929cff9640 1 -- 172.21.15.12:0/3054632332 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9290007070 con 0x7f92880039e0 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.085+0000 7f929cff9640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928c06c8f0 0x7f928c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.086+0000 7f929cff9640 1 -- 172.21.15.12:0/3054632332 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f92900acdc0 con 0x7f92880039e0 2024-04-15T15:21:39.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.086+0000 7f929f7fe640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928c06c8f0 0x7f928c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.086+0000 7f929f7fe640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928c06c8f0 0x7f928c06edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f92a01f83a0 tx=0x7f9294009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.090+0000 7f929cff9640 1 -- 172.21.15.12:0/3054632332 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f929007a940 con 0x7f92880039e0 2024-04-15T15:21:39.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.274+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f92a006ff10 con 0x7f92880039e0 2024-04-15T15:21:39.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.274+0000 7f929cff9640 1 -- 172.21.15.12:0/3054632332 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f929007ec40 con 0x7f92880039e0 2024-04-15T15:21:39.276 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928c06c8f0 msgr2=0x7f928c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928c06c8f0 0x7f928c06edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f92a01f83a0 tx=0x7f9294009040 comp rx=0 tx=0).stop 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 msgr2=0x7f92a01560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f92a01560e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f929000c330 tx=0x7f92a00b43a0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 shutdown_connections 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928c06c8f0 0x7f928c06edb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a01ea4b0 0x7f92a01ea930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92a0142650 0x7f92a01580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 --2- 172.21.15.12:0/3054632332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92880039e0 0x7f92a01560e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 >> 172.21.15.12:0/3054632332 conn(0x7f92a0073360 msgr2=0x7f92a006e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 shutdown_connections 2024-04-15T15:21:39.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.276+0000 7f92a62ce640 1 -- 172.21.15.12:0/3054632332 wait complete. 2024-04-15T15:21:39.289 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:39.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.394+0000 7fc9106d9640 1 Processor -- start 2024-04-15T15:21:39.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.394+0000 7fc9106d9640 1 -- start start 2024-04-15T15:21:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.395+0000 7fc9106d9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc9080cf600 0x7fc9080cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.395+0000 7fc9106d9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9080ae440 con 0x7fc90815ed00 2024-04-15T15:21:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.395+0000 7fc9106d9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9080ae720 con 0x7fc9080af4e0 2024-04-15T15:21:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.395+0000 7fc9106d9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc908071030 con 0x7fc9080cf600 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.395+0000 7fc90e475640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc90815ed00 0x7fc908070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56430/0 (socket says 172.21.15.12:56430) 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.395+0000 7fc90e475640 1 -- 172.21.15.12:0/2645108753 learned_addr learned my addr 172.21.15.12:0/2645108753 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.395+0000 7fc90dc74640 1 --2- 172.21.15.12:0/2645108753 >> v2:172.21.15.55:3300/0 conn(0x7fc9080cf600 0x7fc9080cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.396+0000 7fc90d473640 1 -- 172.21.15.12:0/2645108753 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1798045829 0 0) 0x7fc9080ae440 con 0x7fc90815ed00 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.396+0000 7fc90d473640 1 -- 172.21.15.12:0/2645108753 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8f0003520 con 0x7fc90815ed00 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.396+0000 7fc90d473640 1 -- 172.21.15.12:0/2645108753 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3085964296 0 0) 0x7fc9080ae720 con 0x7fc9080af4e0 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.396+0000 7fc90d473640 1 -- 172.21.15.12:0/2645108753 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9080ae440 con 0x7fc9080af4e0 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.396+0000 7fc90d473640 1 -- 172.21.15.12:0/2645108753 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 272381147 0 0) 0x7fc8f0003520 con 0x7fc90815ed00 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.396+0000 7fc90dc74640 1 -- 172.21.15.12:0/2645108753 >> v1:172.21.15.55:6789/0 conn(0x7fc9080af4e0 legacy=0x7fc9080af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.396+0000 7fc90dc74640 1 -- 172.21.15.12:0/2645108753 >> v1:172.21.15.12:6789/0 conn(0x7fc90815ed00 legacy=0x7fc908070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.396+0000 7fc90dc74640 1 -- 172.21.15.12:0/2645108753 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9080aea00 con 0x7fc9080cf600 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc90dc74640 1 --2- 172.21.15.12:0/2645108753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc9080cf600 0x7fc9080cfa90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0097a0 tx=0x7fc8fc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b913ec6eb1caa83 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc90d473640 1 -- 172.21.15.12:0/2645108753 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc038070 con 0x7fc9080cf600 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc90d473640 1 -- 172.21.15.12:0/2645108753 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc031500 con 0x7fc9080cf600 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc90d473640 1 -- 172.21.15.12:0/2645108753 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80019f0 con 0x7fc9080cf600 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc9106d9640 1 -- 172.21.15.12:0/2645108753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc9080cf600 msgr2=0x7fc9080cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc9106d9640 1 --2- 172.21.15.12:0/2645108753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc9080cf600 0x7fc9080cfa90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0097a0 tx=0x7fc8fc02b820 comp rx=0 tx=0).stop 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc9106d9640 1 -- 172.21.15.12:0/2645108753 shutdown_connections 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc9106d9640 1 --2- 172.21.15.12:0/2645108753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc9080cf600 0x7fc9080cfa90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.397+0000 7fc9106d9640 1 -- 172.21.15.12:0/2645108753 >> 172.21.15.12:0/2645108753 conn(0x7fc90809a9d0 msgr2=0x7fc9080d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.398+0000 7fc9106d9640 1 -- 172.21.15.12:0/2645108753 shutdown_connections 2024-04-15T15:21:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.398+0000 7fc9106d9640 1 -- 172.21.15.12:0/2645108753 wait complete. 2024-04-15T15:21:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc9106d9640 1 Processor -- start 2024-04-15T15:21:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc9106d9640 1 -- start start 2024-04-15T15:21:39.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc9106d9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc9080af4e0 0x7fc9080c4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc9106d9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9080cf600 0x7fc9081f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc9106d9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc90815ed00 0x7fc9080b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc9106d9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc9080701a0 con 0x7fc9080cf600 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc9106d9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc9080aefc0 con 0x7fc90815ed00 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc9106d9640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc9080b0020 con 0x7fc9080af4e0 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc90dc74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9080cf600 0x7fc9081f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc90dc74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9080cf600 0x7fc9081f5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47026/0 (socket says 172.21.15.12:47026) 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.399+0000 7fc90dc74640 1 -- 172.21.15.12:0/38301723 learned_addr learned my addr 172.21.15.12:0/38301723 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90ec76640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc90815ed00 0x7fc9080b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90e475640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc9080af4e0 0x7fc9080c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90dc74640 1 -- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc9080af4e0 msgr2=0x7fc9080c4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90dc74640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc9080af4e0 0x7fc9080c4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90dc74640 1 -- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc90815ed00 msgr2=0x7fc9080b23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90dc74640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc90815ed00 0x7fc9080b23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90dc74640 1 -- 172.21.15.12:0/38301723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9081f6360 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90ec76640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc90815ed00 0x7fc9080b23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc90dc74640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9080cf600 0x7fc9081f5e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc036040 tx=0x7fc8fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc906ffd640 1 -- 172.21.15.12:0/38301723 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc042070 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc906ffd640 1 -- 172.21.15.12:0/38301723 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc03d870 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc90815de80 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc906ffd640 1 -- 172.21.15.12:0/38301723 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc038040 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.400+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9080cc320 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.401+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc908068250 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.402+0000 7fc906ffd640 1 -- 172.21.15.12:0/38301723 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc8fc045070 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.402+0000 7fc906ffd640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8f406c8f0 0x7fc8f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.402+0000 7fc906ffd640 1 -- 172.21.15.12:0/38301723 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc8fc0ad8a0 con 0x7fc9080cf600 2024-04-15T15:21:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.402+0000 7fc90e475640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8f406c8f0 0x7fc8f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.403+0000 7fc90e475640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8f406c8f0 0x7fc8f406edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8001560 tx=0x7fc8f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.407+0000 7fc906ffd640 1 -- 172.21.15.12:0/38301723 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc908068250 con 0x7fc9080cf600 2024-04-15T15:21:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.591+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc9080ce410 con 0x7fc9080cf600 2024-04-15T15:21:39.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.592+0000 7fc906ffd640 1 -- 172.21.15.12:0/38301723 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fc8fc030070 con 0x7fc9080cf600 2024-04-15T15:21:39.593 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.593+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8f406c8f0 msgr2=0x7fc8f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.593+0000 7fc9106d9640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8f406c8f0 0x7fc8f406edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8001560 tx=0x7fc8f8009040 comp rx=0 tx=0).stop 2024-04-15T15:21:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9080cf600 msgr2=0x7fc9081f5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9080cf600 0x7fc9081f5e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc036040 tx=0x7fc8fc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 shutdown_connections 2024-04-15T15:21:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc8f406c8f0 0x7fc8f406edb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc90815ed00 0x7fc9080b23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc9080cf600 0x7fc9081f5e20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 --2- 172.21.15.12:0/38301723 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc9080af4e0 0x7fc9080c4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 >> 172.21.15.12:0/38301723 conn(0x7fc90809a9d0 msgr2=0x7fc9080cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 shutdown_connections 2024-04-15T15:21:39.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.594+0000 7fc9106d9640 1 -- 172.21.15.12:0/38301723 wait complete. 2024-04-15T15:21:39.606 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.713+0000 7f6680d7c640 1 Processor -- start 2024-04-15T15:21:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.713+0000 7f6680d7c640 1 -- start start 2024-04-15T15:21:39.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.714+0000 7f6680d7c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f667c15e700 0x7f667c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.714+0000 7f6680d7c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f667c0b47a0 con 0x7f667c0d2360 2024-04-15T15:21:39.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.714+0000 7f6680d7c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f667c0b4a80 con 0x7f667c0d3270 2024-04-15T15:21:39.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.714+0000 7f6680d7c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f667c0b53b0 con 0x7f667c15e700 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.714+0000 7f6679d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f667c0d2360 0x7f667c0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56442/0 (socket says 172.21.15.12:56442) 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.714+0000 7f6679d9b640 1 -- 172.21.15.12:0/1028884055 learned_addr learned my addr 172.21.15.12:0/1028884055 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3761857630 0 0) 0x7f667c0b47a0 con 0x7f667c0d2360 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667a59c640 1 --2- 172.21.15.12:0/1028884055 >> v2:172.21.15.55:3300/0 conn(0x7f667c15e700 0x7f667c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6668003520 con 0x7f667c0d2360 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3626692172 0 0) 0x7f6668003520 con 0x7f667c0d2360 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 >> v2:172.21.15.55:3300/0 conn(0x7f667c15e700 msgr2=0x7f667c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 --2- 172.21.15.12:0/1028884055 >> v2:172.21.15.55:3300/0 conn(0x7f667c15e700 0x7f667c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 >> v1:172.21.15.55:6789/0 conn(0x7f667c0d3270 legacy=0x7f667c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c0b4d60 con 0x7f667c0d2360 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f6664001670 con 0x7f667c0d2360 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 >> v1:172.21.15.12:6789/0 conn(0x7f667c0d2360 legacy=0x7f667c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680039b0 0x7f6668003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6668004800 0x7f6668004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f666800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f667c0b53b0 con 0x7f66680051c0 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f666800c090 con 0x7f66680039b0 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f666800c290 con 0x7f6668004800 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667ad9d640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f666800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667a59c640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680039b0 0x7f6668003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f6679d9b640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6668004800 0x7f6668004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667ad9d640 1 -- 172.21.15.12:0/1028884055 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6668004800 msgr2=0x7f6668004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667ad9d640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6668004800 0x7f6668004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667ad9d640 1 -- 172.21.15.12:0/1028884055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680039b0 msgr2=0x7f6668003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667ad9d640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680039b0 0x7f6668003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.715+0000 7f667ad9d640 1 -- 172.21.15.12:0/1028884055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6668003520 con 0x7f66680051c0 2024-04-15T15:21:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.716+0000 7f667ad9d640 1 -- 172.21.15.12:0/1028884055 reap_dead start 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.716+0000 7f667ad9d640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f666800b9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6674001750 tx=0x7f6668005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc421ea580cefec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.716+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674001370 con 0x7f66680051c0 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.716+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6674005310 con 0x7f66680051c0 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.716+0000 7f667959a640 1 -- 172.21.15.12:0/1028884055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674003830 con 0x7f66680051c0 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.717+0000 7f6680d7c640 1 -- 172.21.15.12:0/1028884055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 msgr2=0x7f666800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.717+0000 7f6680d7c640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f666800b9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6674001750 tx=0x7f6668005830 comp rx=0 tx=0).stop 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.717+0000 7f6680d7c640 1 -- 172.21.15.12:0/1028884055 shutdown_connections 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.717+0000 7f6680d7c640 1 --2- 172.21.15.12:0/1028884055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f666800b9c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.717+0000 7f6680d7c640 1 -- 172.21.15.12:0/1028884055 >> 172.21.15.12:0/1028884055 conn(0x7f667c08bf60 msgr2=0x7f667c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.717+0000 7f6680d7c640 1 -- 172.21.15.12:0/1028884055 shutdown_connections 2024-04-15T15:21:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.717+0000 7f6680d7c640 1 -- 172.21.15.12:0/1028884055 wait complete. 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.718+0000 7f6680d7c640 1 Processor -- start 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.718+0000 7f6680d7c640 1 -- start start 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f6680d7c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f667c0b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f6680d7c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f667c0cd540 0x7f667c0cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f6680d7c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f667c0d0420 0x7f667c0d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f6680d7c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f667c0d1d40 con 0x7f66680051c0 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f6680d7c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f667c0b4450 con 0x7f667c0d0420 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f6680d7c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f667c0d1ec0 con 0x7f667c0cd540 2024-04-15T15:21:39.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f667c0b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f667c0b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47040/0 (socket says 172.21.15.12:47040) 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667a59c640 1 -- 172.21.15.12:0/1869190625 learned_addr learned my addr 172.21.15.12:0/1869190625 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667ad9d640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f667c0d0420 0x7f667c0d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f6679d9b640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f667c0cd540 0x7f667c0cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667a59c640 1 -- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f667c0cd540 msgr2=0x7f667c0cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667a59c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f667c0cd540 0x7f667c0cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667a59c640 1 -- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f667c0d0420 msgr2=0x7f667c0d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667a59c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f667c0d0420 0x7f667c0d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.719+0000 7f667a59c640 1 -- 172.21.15.12:0/1869190625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c151290 con 0x7f66680051c0 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.720+0000 7f667a59c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f667c0b31a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f666c011e60 tx=0x7f666c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.720+0000 7f6672ffd640 1 -- 172.21.15.12:0/1869190625 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c015630 con 0x7f66680051c0 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.720+0000 7f6672ffd640 1 -- 172.21.15.12:0/1869190625 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666c0157d0 con 0x7f66680051c0 2024-04-15T15:21:39.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.720+0000 7f6672ffd640 1 -- 172.21.15.12:0/1869190625 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c015ba0 con 0x7f66680051c0 2024-04-15T15:21:39.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.720+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667c151570 con 0x7f66680051c0 2024-04-15T15:21:39.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.720+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f667c151b30 con 0x7f66680051c0 2024-04-15T15:21:39.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.721+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667c0689e0 con 0x7f66680051c0 2024-04-15T15:21:39.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.721+0000 7f6672ffd640 1 -- 172.21.15.12:0/1869190625 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f666c01d400 con 0x7f66680051c0 2024-04-15T15:21:39.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.722+0000 7f6672ffd640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f665c06c8f0 0x7f665c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:39.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.722+0000 7f6672ffd640 1 -- 172.21.15.12:0/1869190625 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f666c08cf10 con 0x7f66680051c0 2024-04-15T15:21:39.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.722+0000 7f6679d9b640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f665c06c8f0 0x7f665c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:39.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.722+0000 7f6679d9b640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f665c06c8f0 0x7f665c06edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f667c1566e0 tx=0x7f66640078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:39.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.726+0000 7f6672ffd640 1 -- 172.21.15.12:0/1869190625 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f666c019030 con 0x7f66680051c0 2024-04-15T15:21:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.913+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f667c0ce460 con 0x7f66680051c0 2024-04-15T15:21:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.913+0000 7f6672ffd640 1 -- 172.21.15.12:0/1869190625 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f666c010040 con 0x7f66680051c0 2024-04-15T15:21:39.915 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f665c06c8f0 msgr2=0x7f665c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f665c06c8f0 0x7f665c06edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f667c1566e0 tx=0x7f66640078f0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 msgr2=0x7f667c0b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f667c0b31a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f666c011e60 tx=0x7f666c00f580 comp rx=0 tx=0).stop 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 shutdown_connections 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f665c06c8f0 0x7f665c06edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f667c0d0420 0x7f667c0d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f667c0cd540 0x7f667c0cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 --2- 172.21.15.12:0/1869190625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66680051c0 0x7f667c0b31a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 >> 172.21.15.12:0/1869190625 conn(0x7f667c08bf60 msgr2=0x7f667c0ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 shutdown_connections 2024-04-15T15:21:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:39.915+0000 7f6680d7c640 1 -- 172.21.15.12:0/1869190625 wait complete. 2024-04-15T15:21:39.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:21:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.031+0000 7f07f6f74640 1 Processor -- start 2024-04-15T15:21:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.031+0000 7f07f6f74640 1 -- start start 2024-04-15T15:21:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.031+0000 7f07f6f74640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07f015e790 0x7f07f00b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.031+0000 7f07f6f74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 0x7f07f00d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.031+0000 7f07f6f74640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f015eb70 con 0x7f07f00b7be0 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.031+0000 7f07f6f74640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f07f00d3790 con 0x7f07f015e790 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.031+0000 7f07f6f74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f07f00d45b0 con 0x7f07f00d1cf0 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.032+0000 7f07f4d10640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07f015e790 0x7f07f00b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.032+0000 7f07f4d10640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f07f015e790 0x7f07f00b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47056/0 (socket says 172.21.15.12:47056) 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.032+0000 7f07f4d10640 1 -- 172.21.15.12:0/3908145460 learned_addr learned my addr 172.21.15.12:0/3908145460 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.032+0000 7f07f5511640 1 --2- 172.21.15.12:0/3908145460 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 0x7f07f00d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.032+0000 7f07f4d10640 1 -- 172.21.15.12:0/3908145460 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 msgr2=0x7f07f00d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.032+0000 7f07f4d10640 1 --2- 172.21.15.12:0/3908145460 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 0x7f07f00d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.032+0000 7f07f4d10640 1 -- 172.21.15.12:0/3908145460 >> v1:172.21.15.55:6789/0 conn(0x7f07f00b7be0 legacy=0x7f07f00d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.032+0000 7f07f4d10640 1 -- 172.21.15.12:0/3908145460 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f00d3c80 con 0x7f07f015e790 2024-04-15T15:21:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f4d10640 1 --2- 172.21.15.12:0/3908145460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f00b76a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07f00d3910 tx=0x7f07e402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226581bbe2bce4e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07ef7fe640 1 -- 172.21.15.12:0/3908145460 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e40317c0 con 0x7f07f015e790 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07ef7fe640 1 -- 172.21.15.12:0/3908145460 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e4037070 con 0x7f07f015e790 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07ef7fe640 1 -- 172.21.15.12:0/3908145460 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4031b80 con 0x7f07f015e790 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f6f74640 1 -- 172.21.15.12:0/3908145460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 msgr2=0x7f07f00b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f6f74640 1 --2- 172.21.15.12:0/3908145460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f00b76a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07f00d3910 tx=0x7f07e402b860 comp rx=0 tx=0).stop 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f6f74640 1 -- 172.21.15.12:0/3908145460 shutdown_connections 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f6f74640 1 --2- 172.21.15.12:0/3908145460 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 0x7f07f00d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f6f74640 1 --2- 172.21.15.12:0/3908145460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f00b76a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f6f74640 1 -- 172.21.15.12:0/3908145460 >> 172.21.15.12:0/3908145460 conn(0x7f07f0067b40 msgr2=0x7f07f0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f6f74640 1 -- 172.21.15.12:0/3908145460 shutdown_connections 2024-04-15T15:21:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.033+0000 7f07f6f74640 1 -- 172.21.15.12:0/3908145460 wait complete. 2024-04-15T15:21:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.035+0000 7f07f6f74640 1 Processor -- start 2024-04-15T15:21:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.035+0000 7f07f6f74640 1 -- start start 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.035+0000 7f07f6f74640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07f00b7be0 0x7f07f01508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.035+0000 7f07f6f74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 0x7f07f0160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.035+0000 7f07f6f74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f01612c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.035+0000 7f07f6f74640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f07f00d4f80 con 0x7f07f015e790 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.035+0000 7f07f6f74640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f07f00d4240 con 0x7f07f00b7be0 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.035+0000 7f07f6f74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f07f00d5100 con 0x7f07f00d1cf0 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f01612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f01612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47062/0 (socket says 172.21.15.12:47062) 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 -- 172.21.15.12:0/2575088634 learned_addr learned my addr 172.21.15.12:0/2575088634 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f4d10640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07f00b7be0 0x7f07f01508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07effff640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 0x7f07f0160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 -- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 msgr2=0x7f07f0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 0x7f07f0160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 -- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07f00b7be0 msgr2=0x7f07f01508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07f00b7be0 0x7f07f01508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 -- 172.21.15.12:0/2575088634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f00cd860 con 0x7f07f015e790 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f5511640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f01612c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f07e800efc0 tx=0x7f07e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07ed7fa640 1 -- 172.21.15.12:0/2575088634 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e8018020 con 0x7f07f015e790 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07ed7fa640 1 -- 172.21.15.12:0/2575088634 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e8009b10 con 0x7f07f015e790 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f00cdb40 con 0x7f07f015e790 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07ed7fa640 1 -- 172.21.15.12:0/2575088634 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e80146c0 con 0x7f07f015e790 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.036+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f00ce100 con 0x7f07f015e790 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.037+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f00b9260 con 0x7f07f015e790 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.038+0000 7f07ed7fa640 1 -- 172.21.15.12:0/2575088634 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f07e8005d10 con 0x7f07f015e790 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.038+0000 7f07ed7fa640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f07d406c8f0 0x7f07d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.038+0000 7f07ed7fa640 1 -- 172.21.15.12:0/2575088634 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f07e808c870 con 0x7f07f015e790 2024-04-15T15:21:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.039+0000 7f07f4d10640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f07d406c8f0 0x7f07d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.039+0000 7f07f4d10640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f07d406c8f0 0x7f07d406edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f07e4036000 tx=0x7f07e403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.043+0000 7f07ed7fa640 1 -- 172.21.15.12:0/2575088634 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f07f00b9260 con 0x7f07f015e790 2024-04-15T15:21:40.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.268+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07f00689e0 con 0x7f07f015e790 2024-04-15T15:21:40.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.268+0000 7f07ed7fa640 1 -- 172.21.15.12:0/2575088634 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f07e8022070 con 0x7f07f015e790 2024-04-15T15:21:40.270 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:40.271 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:33.113220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:21:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-15T15:21:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f07d406c8f0 msgr2=0x7f07d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f07d406c8f0 0x7f07d406edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f07e4036000 tx=0x7f07e403d040 comp rx=0 tx=0).stop 2024-04-15T15:21:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 msgr2=0x7f07f01612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f01612c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f07e800efc0 tx=0x7f07e800c490 comp rx=0 tx=0).stop 2024-04-15T15:21:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 shutdown_connections 2024-04-15T15:21:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f07f015e790 0x7f07f01612c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f07f00d1cf0 0x7f07f0160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f07d406c8f0 0x7f07d406edb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 --2- 172.21.15.12:0/2575088634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07f00b7be0 0x7f07f01508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 >> 172.21.15.12:0/2575088634 conn(0x7f07f0067b40 msgr2=0x7f07f00d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 shutdown_connections 2024-04-15T15:21:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.270+0000 7f07f6f74640 1 -- 172.21.15.12:0/2575088634 wait complete. 2024-04-15T15:21:40.283 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:40.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.386+0000 7fa5c6a14640 1 Processor -- start 2024-04-15T15:21:40.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.387+0000 7fa5c6a14640 1 -- start start 2024-04-15T15:21:40.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.387+0000 7fa5c6a14640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5c00b0560 0x7fa5c006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.387+0000 7fa5c6a14640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c00b09a0 0x7fa5c006bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.387+0000 7fa5c6a14640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5c00cbd10 con 0x7fa5c015eb00 2024-04-15T15:21:40.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.387+0000 7fa5c6a14640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa5c015eee0 con 0x7fa5c00b0560 2024-04-15T15:21:40.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.387+0000 7fa5c6a14640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c00cbff0 con 0x7fa5c00b09a0 2024-04-15T15:21:40.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.387+0000 7fa5b7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5c00b0560 0x7fa5c006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.388+0000 7fa5b7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa5c00b0560 0x7fa5c006b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47076/0 (socket says 172.21.15.12:47076) 2024-04-15T15:21:40.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.388+0000 7fa5b7fff640 1 -- 172.21.15.12:0/4026983689 learned_addr learned my addr 172.21.15.12:0/4026983689 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.388+0000 7fa5c4fb1640 1 --2- 172.21.15.12:0/4026983689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c00b09a0 0x7fa5c006bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.388+0000 7fa5b7fff640 1 -- 172.21.15.12:0/4026983689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c00b09a0 msgr2=0x7fa5c006bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.388+0000 7fa5b7fff640 1 --2- 172.21.15.12:0/4026983689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c00b09a0 0x7fa5c006bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.388+0000 7fa5b7fff640 1 -- 172.21.15.12:0/4026983689 >> v1:172.21.15.55:6789/0 conn(0x7fa5c015eb00 legacy=0x7fa5c00ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.388+0000 7fa5b7fff640 1 -- 172.21.15.12:0/4026983689 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c00cc2d0 con 0x7fa5c00b0560 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5b7fff640 1 --2- 172.21.15.12:0/4026983689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b0560 0x7fa5c006b9e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009870 tx=0x7fa5a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a086de6dead10600 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5bf7fe640 1 -- 172.21.15.12:0/4026983689 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8034070 con 0x7fa5c00b0560 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5bf7fe640 1 -- 172.21.15.12:0/4026983689 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a8030840 con 0x7fa5c00b0560 2024-04-15T15:21:40.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5bf7fe640 1 -- 172.21.15.12:0/4026983689 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8030c10 con 0x7fa5c00b0560 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5c6a14640 1 -- 172.21.15.12:0/4026983689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b0560 msgr2=0x7fa5c006b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/4026983689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b0560 0x7fa5c006b9e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009870 tx=0x7fa5a802b980 comp rx=0 tx=0).stop 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5c6a14640 1 -- 172.21.15.12:0/4026983689 shutdown_connections 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/4026983689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b0560 0x7fa5c006b9e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/4026983689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c00b09a0 0x7fa5c006bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5c6a14640 1 -- 172.21.15.12:0/4026983689 >> 172.21.15.12:0/4026983689 conn(0x7fa5c009a9d0 msgr2=0x7fa5c00d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5c6a14640 1 -- 172.21.15.12:0/4026983689 shutdown_connections 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.389+0000 7fa5c6a14640 1 -- 172.21.15.12:0/4026983689 wait complete. 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5c6a14640 1 Processor -- start 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5c6a14640 1 -- start start 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5c6a14640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5c00b0560 0x7fa5c00b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5c6a14640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b09a0 0x7fa5c0152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5c6a14640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c015eb00 0x7fa5c00b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5c6a14640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c00cec60 con 0x7fa5c00b09a0 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5c6a14640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c00cc890 con 0x7fa5c00b0560 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5c6a14640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c00cede0 con 0x7fa5c015eb00 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5b7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b09a0 0x7fa5c0152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5b7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b09a0 0x7fa5c0152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47080/0 (socket says 172.21.15.12:47080) 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.391+0000 7fa5b7fff640 1 -- 172.21.15.12:0/3771294629 learned_addr learned my addr 172.21.15.12:0/3771294629 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5bffff640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5c00b0560 0x7fa5c00b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5c4fb1640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c015eb00 0x7fa5c00b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5b7fff640 1 -- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c015eb00 msgr2=0x7fa5c00b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5b7fff640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c015eb00 0x7fa5c00b5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5b7fff640 1 -- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5c00b0560 msgr2=0x7fa5c00b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5b7fff640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5c00b0560 0x7fa5c00b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5b7fff640 1 -- 172.21.15.12:0/3771294629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c015c010 con 0x7fa5c00b09a0 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5c4fb1640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c015eb00 0x7fa5c00b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5b7fff640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b09a0 0x7fa5c0152910 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8000c00 tx=0x7fa5a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5bd7fa640 1 -- 172.21.15.12:0/3771294629 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8034040 con 0x7fa5c00b09a0 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5bd7fa640 1 -- 172.21.15.12:0/3771294629 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a8039d90 con 0x7fa5c00b09a0 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c00b6c20 con 0x7fa5c00b09a0 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5bd7fa640 1 -- 172.21.15.12:0/3771294629 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8043610 con 0x7fa5c00b09a0 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.392+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c00b71e0 con 0x7fa5c00b09a0 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.393+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c00690f0 con 0x7fa5c00b09a0 2024-04-15T15:21:40.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.394+0000 7fa5bd7fa640 1 -- 172.21.15.12:0/3771294629 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa5a804a060 con 0x7fa5c00b09a0 2024-04-15T15:21:40.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.394+0000 7fa5bd7fa640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa5a006c8f0 0x7fa5a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.394+0000 7fa5bd7fa640 1 -- 172.21.15.12:0/3771294629 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa5a80ade60 con 0x7fa5c00b09a0 2024-04-15T15:21:40.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.395+0000 7fa5bffff640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa5a006c8f0 0x7fa5a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.395+0000 7fa5bffff640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa5a006c8f0 0x7fa5a006edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0001640 tx=0x7fa5b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.399+0000 7fa5bd7fa640 1 -- 172.21.15.12:0/3771294629 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5a803e070 con 0x7fa5c00b09a0 2024-04-15T15:21:40.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.585+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5c00b1240 con 0x7fa5c00b09a0 2024-04-15T15:21:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.586+0000 7fa5bd7fa640 1 -- 172.21.15.12:0/3771294629 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7fa5a8080e20 con 0x7fa5c00b09a0 2024-04-15T15:21:40.588 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:40.588 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa5a006c8f0 msgr2=0x7fa5a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa5a006c8f0 0x7fa5a006edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0001640 tx=0x7fa5b0009040 comp rx=0 tx=0).stop 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b09a0 msgr2=0x7fa5c0152910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b09a0 0x7fa5c0152910 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8000c00 tx=0x7fa5a80046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 shutdown_connections 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa5c015eb00 0x7fa5c00b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c00b09a0 0x7fa5c0152910 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa5a006c8f0 0x7fa5a006edb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 --2- 172.21.15.12:0/3771294629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5c00b0560 0x7fa5c00b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 >> 172.21.15.12:0/3771294629 conn(0x7fa5c009a9d0 msgr2=0x7fa5c015da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 shutdown_connections 2024-04-15T15:21:40.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.588+0000 7fa5c6a14640 1 -- 172.21.15.12:0/3771294629 wait complete. 2024-04-15T15:21:40.601 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:40.602 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:40.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.707+0000 7faddab00640 1 Processor -- start 2024-04-15T15:21:40.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.707+0000 7faddab00640 1 -- start start 2024-04-15T15:21:40.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7faddab00640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fadd40d1610 0x7fadd40b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7faddab00640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadd40d2480 con 0x7fadd40d3240 2024-04-15T15:21:40.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7faddab00640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadd40d2760 con 0x7fadd415e790 2024-04-15T15:21:40.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7faddab00640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fadd40d2db0 con 0x7fadd40d1610 2024-04-15T15:21:40.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7fadd3fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fadd40d3240 0x7fadd40d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56454/0 (socket says 172.21.15.12:56454) 2024-04-15T15:21:40.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7fadd3fff640 1 -- 172.21.15.12:0/2879999407 learned_addr learned my addr 172.21.15.12:0/2879999407 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:40.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7fadd909d640 1 --2- 172.21.15.12:0/2879999407 >> v2:172.21.15.55:3300/0 conn(0x7fadd40d1610 0x7fadd40b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7fadd37fe640 1 -- 172.21.15.12:0/2879999407 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3416309919 0 0) 0x7fadd40d2480 con 0x7fadd40d3240 2024-04-15T15:21:40.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.708+0000 7fadd37fe640 1 -- 172.21.15.12:0/2879999407 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadbc003520 con 0x7fadd40d3240 2024-04-15T15:21:40.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.709+0000 7fadd909d640 1 -- 172.21.15.12:0/2879999407 >> v1:172.21.15.55:6789/0 conn(0x7fadd415e790 legacy=0x7fadd40cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.709+0000 7fadd37fe640 1 -- 172.21.15.12:0/2879999407 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475142932 0 0) 0x7fadd40d2760 con 0x7fadd415e790 2024-04-15T15:21:40.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.709+0000 7fadd909d640 1 -- 172.21.15.12:0/2879999407 >> v1:172.21.15.12:6789/0 conn(0x7fadd40d3240 legacy=0x7fadd40d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.709+0000 7fadd909d640 1 -- 172.21.15.12:0/2879999407 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd40d2a40 con 0x7fadd40d1610 2024-04-15T15:21:40.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.709+0000 7fadd909d640 1 --2- 172.21.15.12:0/2879999407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d1610 0x7fadd40b74f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fadd40d2f30 tx=0x7fadc402b760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee864c33abd3a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7fadd37fe640 1 -- 172.21.15.12:0/2879999407 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4037070 con 0x7fadd40d1610 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7fadd37fe640 1 -- 172.21.15.12:0/2879999407 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc4030510 con 0x7fadd40d1610 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7fadd37fe640 1 -- 172.21.15.12:0/2879999407 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8001670 con 0x7fadd40d1610 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7faddab00640 1 -- 172.21.15.12:0/2879999407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d1610 msgr2=0x7fadd40b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7faddab00640 1 --2- 172.21.15.12:0/2879999407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d1610 0x7fadd40b74f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fadd40d2f30 tx=0x7fadc402b760 comp rx=0 tx=0).stop 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7faddab00640 1 -- 172.21.15.12:0/2879999407 shutdown_connections 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7faddab00640 1 --2- 172.21.15.12:0/2879999407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d1610 0x7fadd40b74f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7faddab00640 1 -- 172.21.15.12:0/2879999407 >> 172.21.15.12:0/2879999407 conn(0x7fadd408bf60 msgr2=0x7fadd40d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7faddab00640 1 -- 172.21.15.12:0/2879999407 shutdown_connections 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.710+0000 7faddab00640 1 -- 172.21.15.12:0/2879999407 wait complete. 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.711+0000 7faddab00640 1 Processor -- start 2024-04-15T15:21:40.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.711+0000 7faddab00640 1 -- start start 2024-04-15T15:21:40.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7faddab00640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fadd40d1610 0x7fadd414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7faddab00640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d3240 0x7fadd40c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7faddab00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadd415e790 0x7fadd40b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7faddab00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40d5180 con 0x7fadd415e790 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7faddab00640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40d0430 con 0x7fadd40d3240 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7faddab00640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fadd40d5300 con 0x7fadd40d1610 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7fadd909d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadd415e790 0x7fadd40b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7fadd909d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadd415e790 0x7fadd40b46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47090/0 (socket says 172.21.15.12:47090) 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.712+0000 7fadd909d640 1 -- 172.21.15.12:0/2974338319 learned_addr learned my addr 172.21.15.12:0/2974338319 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd3fff640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d3240 0x7fadd40c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd889c640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fadd40d1610 0x7fadd414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd909d640 1 -- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fadd40d1610 msgr2=0x7fadd414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd909d640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fadd40d1610 0x7fadd414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd909d640 1 -- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d3240 msgr2=0x7fadd40c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd909d640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d3240 0x7fadd40c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd909d640 1 -- 172.21.15.12:0/2974338319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd40c86d0 con 0x7fadd415e790 2024-04-15T15:21:40.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd909d640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadd415e790 0x7fadd40b46b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fadc402b7a0 tx=0x7fadc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd17fa640 1 -- 172.21.15.12:0/2974338319 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4042020 con 0x7fadd415e790 2024-04-15T15:21:40.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd17fa640 1 -- 172.21.15.12:0/2974338319 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc40044c0 con 0x7fadd415e790 2024-04-15T15:21:40.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd40c8c90 con 0x7fadd415e790 2024-04-15T15:21:40.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7fadd17fa640 1 -- 172.21.15.12:0/2974338319 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4037040 con 0x7fadd415e790 2024-04-15T15:21:40.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.713+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd40c9250 con 0x7fadd415e790 2024-04-15T15:21:40.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.714+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd40689e0 con 0x7fadd415e790 2024-04-15T15:21:40.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.715+0000 7fadd17fa640 1 -- 172.21.15.12:0/2974338319 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fadc4045020 con 0x7fadd415e790 2024-04-15T15:21:40.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.715+0000 7fadd17fa640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadc006c8f0 0x7fadc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:40.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.715+0000 7fadd17fa640 1 -- 172.21.15.12:0/2974338319 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fadc40ada80 con 0x7fadd415e790 2024-04-15T15:21:40.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.715+0000 7fadd889c640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadc006c8f0 0x7fadc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:40.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.716+0000 7fadd889c640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadc006c8f0 0x7fadc006edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0012e0 tx=0x7fadcc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:40.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.720+0000 7fadd17fa640 1 -- 172.21.15.12:0/2974338319 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fadc404c030 con 0x7fadd415e790 2024-04-15T15:21:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.904+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fadd40d1ce0 con 0x7fadd415e790 2024-04-15T15:21:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.904+0000 7fadd17fa640 1 -- 172.21.15.12:0/2974338319 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fadc407b5b0 con 0x7fadd415e790 2024-04-15T15:21:40.906 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadc006c8f0 msgr2=0x7fadc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadc006c8f0 0x7fadc006edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0012e0 tx=0x7fadcc009040 comp rx=0 tx=0).stop 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadd415e790 msgr2=0x7fadd40b46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadd415e790 0x7fadd40b46b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fadc402b7a0 tx=0x7fadc40046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 shutdown_connections 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadd415e790 0x7fadd40b46b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fadc006c8f0 0x7fadc006edb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadd40d3240 0x7fadd40c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 --2- 172.21.15.12:0/2974338319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fadd40d1610 0x7fadd414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 >> 172.21.15.12:0/2974338319 conn(0x7fadd408bf60 msgr2=0x7fadd40cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 shutdown_connections 2024-04-15T15:21:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:40.906+0000 7faddab00640 1 -- 172.21.15.12:0/2974338319 wait complete. 2024-04-15T15:21:40.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:41.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.024+0000 7fd339fe9640 1 Processor -- start 2024-04-15T15:21:41.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.024+0000 7fd339fe9640 1 -- start start 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd339fe9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd33406e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd339fe9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd3340b4ab0 0x7fd33409b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd339fe9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd33406e9d0 con 0x7fd33409a9d0 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd339fe9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd33415ed70 con 0x7fd3340b4ab0 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd339fe9640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd33406e000 con 0x7fd33415e990 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd332ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd3340b4ab0 0x7fd33409b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd332ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd3340b4ab0 0x7fd33409b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47094/0 (socket says 172.21.15.12:47094) 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd333fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd33409a9d0 0x7fd3340cca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56468/0 (socket says 172.21.15.12:56468) 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.025+0000 7fd332ffd640 1 -- 172.21.15.12:0/2646607873 learned_addr learned my addr 172.21.15.12:0/2646607873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.026+0000 7fd3337fe640 1 --2- 172.21.15.12:0/2646607873 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd33406e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.026+0000 7fd332ffd640 1 -- 172.21.15.12:0/2646607873 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 msgr2=0x7fd33406e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.026+0000 7fd332ffd640 1 --2- 172.21.15.12:0/2646607873 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd33406e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.026+0000 7fd332ffd640 1 -- 172.21.15.12:0/2646607873 >> v1:172.21.15.12:6789/0 conn(0x7fd33409a9d0 legacy=0x7fd3340cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.026+0000 7fd332ffd640 1 -- 172.21.15.12:0/2646607873 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33406d3f0 con 0x7fd3340b4ab0 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.026+0000 7fd3327fc640 1 -- 172.21.15.12:0/2646607873 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259474032 0 0) 0x7fd33406e9d0 con 0x7fd33409a9d0 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.026+0000 7fd3337fe640 1 --2- 172.21.15.12:0/2646607873 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd33406e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.026+0000 7fd332ffd640 1 --2- 172.21.15.12:0/2646607873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd33409b720 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd328009870 tx=0x7fd32802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad87fefec0e1ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd3327fc640 1 -- 172.21.15.12:0/2646607873 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd328034070 con 0x7fd3340b4ab0 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd3327fc640 1 -- 172.21.15.12:0/2646607873 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd328030840 con 0x7fd3340b4ab0 2024-04-15T15:21:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd3327fc640 1 -- 172.21.15.12:0/2646607873 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd328030c10 con 0x7fd3340b4ab0 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd339fe9640 1 -- 172.21.15.12:0/2646607873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 msgr2=0x7fd33409b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2646607873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd33409b720 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd328009870 tx=0x7fd32802b980 comp rx=0 tx=0).stop 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd339fe9640 1 -- 172.21.15.12:0/2646607873 shutdown_connections 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2646607873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd33409b720 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2646607873 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd33406e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd339fe9640 1 -- 172.21.15.12:0/2646607873 >> 172.21.15.12:0/2646607873 conn(0x7fd3340b3eb0 msgr2=0x7fd3340b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd339fe9640 1 -- 172.21.15.12:0/2646607873 shutdown_connections 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.027+0000 7fd339fe9640 1 -- 172.21.15.12:0/2646607873 wait complete. 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.028+0000 7fd339fe9640 1 Processor -- start 2024-04-15T15:21:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd339fe9640 1 -- start start 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd339fe9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd33409a9d0 0x7fd3341ee1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd339fe9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd334070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd339fe9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd334070a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd339fe9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd33406dc90 con 0x7fd3340b4ab0 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd339fe9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd33406d9b0 con 0x7fd33409a9d0 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd339fe9640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd33409bc60 con 0x7fd33415e990 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd332ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd334070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd332ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd334070550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47104/0 (socket says 172.21.15.12:47104) 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd332ffd640 1 -- 172.21.15.12:0/2294467414 learned_addr learned my addr 172.21.15.12:0/2294467414 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.029+0000 7fd333fff640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd334070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd3337fe640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd33409a9d0 0x7fd3341ee1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd332ffd640 1 -- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 msgr2=0x7fd334070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd332ffd640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd334070a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd332ffd640 1 -- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd33409a9d0 msgr2=0x7fd3341ee1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd332ffd640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd33409a9d0 0x7fd3341ee1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd332ffd640 1 -- 172.21.15.12:0/2294467414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3341f7a40 con 0x7fd3340b4ab0 2024-04-15T15:21:41.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd332ffd640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd334070550 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd32802b9c0 tx=0x7fd328004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd31bfff640 1 -- 172.21.15.12:0/2294467414 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd328048070 con 0x7fd3340b4ab0 2024-04-15T15:21:41.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd31bfff640 1 -- 172.21.15.12:0/2294467414 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd328030640 con 0x7fd3340b4ab0 2024-04-15T15:21:41.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3341f87c0 con 0x7fd3340b4ab0 2024-04-15T15:21:41.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd31bfff640 1 -- 172.21.15.12:0/2294467414 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd328034040 con 0x7fd3340b4ab0 2024-04-15T15:21:41.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.030+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3341f7fc0 con 0x7fd3340b4ab0 2024-04-15T15:21:41.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.031+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3340689e0 con 0x7fd3340b4ab0 2024-04-15T15:21:41.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.031+0000 7fd31bfff640 1 -- 172.21.15.12:0/2294467414 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd328037070 con 0x7fd3340b4ab0 2024-04-15T15:21:41.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.032+0000 7fd31bfff640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd32006c8f0 0x7fd32006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.032+0000 7fd31bfff640 1 -- 172.21.15.12:0/2294467414 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd3280add20 con 0x7fd3340b4ab0 2024-04-15T15:21:41.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.032+0000 7fd3337fe640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd32006c8f0 0x7fd32006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.032+0000 7fd3337fe640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd32006c8f0 0x7fd32006edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd3240075b0 tx=0x7fd324007540 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.036+0000 7fd31bfff640 1 -- 172.21.15.12:0/2294467414 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd32807be20 con 0x7fd3340b4ab0 2024-04-15T15:21:41.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.223+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd3340d0ab0 con 0x7fd3340b4ab0 2024-04-15T15:21:41.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.224+0000 7fd31bfff640 1 -- 172.21.15.12:0/2294467414 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fd32807b720 con 0x7fd3340b4ab0 2024-04-15T15:21:41.225 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd32006c8f0 msgr2=0x7fd32006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd32006c8f0 0x7fd32006edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd3240075b0 tx=0x7fd324007540 comp rx=0 tx=0).stop 2024-04-15T15:21:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 msgr2=0x7fd334070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd334070550 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd32802b9c0 tx=0x7fd328004730 comp rx=0 tx=0).stop 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 shutdown_connections 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd33415e990 0x7fd334070a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3340b4ab0 0x7fd334070550 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd32006c8f0 0x7fd32006edb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 --2- 172.21.15.12:0/2294467414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd33409a9d0 0x7fd3341ee1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 >> 172.21.15.12:0/2294467414 conn(0x7fd3340b3eb0 msgr2=0x7fd334071510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 shutdown_connections 2024-04-15T15:21:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.226+0000 7fd339fe9640 1 -- 172.21.15.12:0/2294467414 wait complete. 2024-04-15T15:21:41.238 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:41.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.344+0000 7f3c0cd59640 1 Processor -- start 2024-04-15T15:21:41.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.344+0000 7f3c0cd59640 1 -- start start 2024-04-15T15:21:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c0cd59640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3c080d5150 0x7f3c080d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c0cd59640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c0815eb70 con 0x7f3c080d15e0 2024-04-15T15:21:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c0cd59640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c080d3c80 con 0x7f3c0815e790 2024-04-15T15:21:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c0cd59640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3c080d3790 con 0x7f3c080d5150 2024-04-15T15:21:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c05d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3c080d15e0 0x7f3c080d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56472/0 (socket says 172.21.15.12:56472) 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c05d9b640 1 -- 172.21.15.12:0/726319687 learned_addr learned my addr 172.21.15.12:0/726319687 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c06d9d640 1 --2- 172.21.15.12:0/726319687 >> v2:172.21.15.55:3300/0 conn(0x7f3c080d5150 0x7f3c080d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340607135 0 0) 0x7f3c0815eb70 con 0x7f3c080d15e0 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.345+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bf4003520 con 0x7f3c080d15e0 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2599693662 0 0) 0x7f3bf4003520 con 0x7f3c080d15e0 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 >> v2:172.21.15.55:3300/0 conn(0x7f3c080d5150 msgr2=0x7f3c080d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 --2- 172.21.15.12:0/726319687 >> v2:172.21.15.55:3300/0 conn(0x7f3c080d5150 0x7f3c080d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 >> v1:172.21.15.55:6789/0 conn(0x7f3c0815e790 legacy=0x7f3c080b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c080d3f60 con 0x7f3c080d15e0 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f3bf0001670 con 0x7f3c080d15e0 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 >> v1:172.21.15.12:6789/0 conn(0x7f3c080d15e0 legacy=0x7f3c080d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40039b0 0x7f3bf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bf4004800 0x7f3bf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bf40052e0 0x7f3bf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3c080d3790 con 0x7f3bf40052e0 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf400c1b0 con 0x7f3bf40039b0 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3bf400c3b0 con 0x7f3bf4004800 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c0659c640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40039b0 0x7f3bf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c06d9d640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bf40052e0 0x7f3bf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c05d9b640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bf4004800 0x7f3bf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c06d9d640 1 -- 172.21.15.12:0/726319687 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bf4004800 msgr2=0x7f3bf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c06d9d640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3bf4004800 0x7f3bf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c06d9d640 1 -- 172.21.15.12:0/726319687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40039b0 msgr2=0x7f3bf4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c06d9d640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40039b0 0x7f3bf4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.346+0000 7f3c06d9d640 1 -- 172.21.15.12:0/726319687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf4003520 con 0x7f3bf40052e0 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.347+0000 7f3c06d9d640 1 -- 172.21.15.12:0/726319687 reap_dead start 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.347+0000 7f3c06d9d640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bf40052e0 0x7f3bf400b9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009a90 tx=0x7f3bf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e64b8af53a098a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.347+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00009690 con 0x7f3bf40052e0 2024-04-15T15:21:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.347+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c00004020 con 0x7f3bf40052e0 2024-04-15T15:21:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.347+0000 7f3c0559a640 1 -- 172.21.15.12:0/726319687 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00004410 con 0x7f3bf40052e0 2024-04-15T15:21:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.348+0000 7f3c0cd59640 1 -- 172.21.15.12:0/726319687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bf40052e0 msgr2=0x7f3bf400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.348+0000 7f3c0cd59640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bf40052e0 0x7f3bf400b9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009a90 tx=0x7f3bf4005830 comp rx=0 tx=0).stop 2024-04-15T15:21:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.348+0000 7f3c0cd59640 1 -- 172.21.15.12:0/726319687 shutdown_connections 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.348+0000 7f3c0cd59640 1 --2- 172.21.15.12:0/726319687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bf40052e0 0x7f3bf400b9c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.348+0000 7f3c0cd59640 1 -- 172.21.15.12:0/726319687 >> 172.21.15.12:0/726319687 conn(0x7f3c08067b40 msgr2=0x7f3c08067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.348+0000 7f3c0cd59640 1 -- 172.21.15.12:0/726319687 shutdown_connections 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.348+0000 7f3c0cd59640 1 -- 172.21.15.12:0/726319687 wait complete. 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.349+0000 7f3c0cd59640 1 Processor -- start 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.349+0000 7f3c0cd59640 1 -- start start 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c0cd59640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40052e0 0x7f3c0814fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c0cd59640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c080c80f0 0x7f3c080c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c0cd59640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3c080cd6f0 0x7f3c080cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c0cd59640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3c080d1a90 con 0x7f3c080c80f0 2024-04-15T15:21:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c0cd59640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3c080d45b0 con 0x7f3bf40052e0 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c0cd59640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3c080d1c10 con 0x7f3c080cd6f0 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c080c80f0 0x7f3c080c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c080c80f0 0x7f3c080c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47112/0 (socket says 172.21.15.12:47112) 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 -- 172.21.15.12:0/3390015024 learned_addr learned my addr 172.21.15.12:0/3390015024 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c0659c640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40052e0 0x7f3c0814fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c06d9d640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3c080cd6f0 0x7f3c080cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 -- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3c080cd6f0 msgr2=0x7f3c080cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3c080cd6f0 0x7f3c080cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 -- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40052e0 msgr2=0x7f3c0814fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40052e0 0x7f3c0814fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 -- 172.21.15.12:0/3390015024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c080c8bb0 con 0x7f3c080c80f0 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3c05d9b640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c080c80f0 0x7f3c080c8570 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0007340 tx=0x7f3bf0007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.350+0000 7f3bfeffd640 1 -- 172.21.15.12:0/3390015024 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000f860 con 0x7f3c080c80f0 2024-04-15T15:21:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.351+0000 7f3bfeffd640 1 -- 172.21.15.12:0/3390015024 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf0018070 con 0x7f3c080c80f0 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.351+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c080c8e90 con 0x7f3c080c80f0 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.351+0000 7f3bfeffd640 1 -- 172.21.15.12:0/3390015024 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000fc60 con 0x7f3c080c80f0 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.351+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c0815c030 con 0x7f3c080c80f0 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.352+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c080b5690 con 0x7f3c080c80f0 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.352+0000 7f3bfeffd640 1 -- 172.21.15.12:0/3390015024 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3bf001d460 con 0x7f3c080c80f0 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.352+0000 7f3bfeffd640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3be806c8f0 0x7f3be806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.353+0000 7f3bfeffd640 1 -- 172.21.15.12:0/3390015024 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3bf008bc80 con 0x7f3c080c80f0 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.353+0000 7f3c0659c640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3be806c8f0 0x7f3be806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.353+0000 7f3c0659c640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3be806c8f0 0x7f3be806edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8000f80 tx=0x7f3bf8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.357+0000 7f3bfeffd640 1 -- 172.21.15.12:0/3390015024 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c080b5690 con 0x7f3c080c80f0 2024-04-15T15:21:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.540+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3c080689e0 con 0x7f3c080c80f0 2024-04-15T15:21:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.541+0000 7f3bfeffd640 1 -- 172.21.15.12:0/3390015024 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f3bf00597b0 con 0x7f3c080c80f0 2024-04-15T15:21:41.542 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3be806c8f0 msgr2=0x7f3be806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3be806c8f0 0x7f3be806edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8000f80 tx=0x7f3bf8009040 comp rx=0 tx=0).stop 2024-04-15T15:21:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c080c80f0 msgr2=0x7f3c080c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c080c80f0 0x7f3c080c8570 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0007340 tx=0x7f3bf0007800 comp rx=0 tx=0).stop 2024-04-15T15:21:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 shutdown_connections 2024-04-15T15:21:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3c080cd6f0 0x7f3c080cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3c080c80f0 0x7f3c080c8570 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3be806c8f0 0x7f3be806edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 --2- 172.21.15.12:0/3390015024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bf40052e0 0x7f3c0814fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 >> 172.21.15.12:0/3390015024 conn(0x7f3c08067b40 msgr2=0x7f3c080b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 shutdown_connections 2024-04-15T15:21:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.543+0000 7f3c0cd59640 1 -- 172.21.15.12:0/3390015024 wait complete. 2024-04-15T15:21:41.555 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:41.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.661+0000 7f3cc45bd640 1 Processor -- start 2024-04-15T15:21:41.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.661+0000 7f3cc45bd640 1 -- start start 2024-04-15T15:21:41.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.662+0000 7f3cc45bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3cbc0d3170 0x7f3cbc06d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.662+0000 7f3cc45bd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cbc0d0530 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.662+0000 7f3cc45bd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cbc0cf490 con 0x7f3cbc0d15c0 2024-04-15T15:21:41.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.662+0000 7f3cc45bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3cbc0cf770 con 0x7f3cbc0d3170 2024-04-15T15:21:41.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.662+0000 7f3cc2b5a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3cbc0d3170 0x7f3cbc06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.662+0000 7f3cc2359640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3cbc15e7a0 0x7f3cbc0d0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56476/0 (socket says 172.21.15.12:56476) 2024-04-15T15:21:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.662+0000 7f3cc2359640 1 -- 172.21.15.12:0/3672028127 learned_addr learned my addr 172.21.15.12:0/3672028127 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.663+0000 7f3cc2b5a640 1 -- 172.21.15.12:0/3672028127 >> v1:172.21.15.55:6789/0 conn(0x7f3cbc0d15c0 legacy=0x7f3cbc06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.663+0000 7f3cc2b5a640 1 -- 172.21.15.12:0/3672028127 >> v1:172.21.15.12:6789/0 conn(0x7f3cbc15e7a0 legacy=0x7f3cbc0d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.663+0000 7f3cc2b5a640 1 -- 172.21.15.12:0/3672028127 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cbc0cfa50 con 0x7f3cbc0d3170 2024-04-15T15:21:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.663+0000 7f3cc1357640 1 -- 172.21.15.12:0/3672028127 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 491341422 0 0) 0x7f3cbc0d0530 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.663+0000 7f3cc2b5a640 1 --2- 172.21.15.12:0/3672028127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc0d3170 0x7f3cbc06d220 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009d90 tx=0x7f3cac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ba32c74dcfd3b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc1357640 1 -- 172.21.15.12:0/3672028127 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac039070 con 0x7f3cbc0d3170 2024-04-15T15:21:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc1357640 1 -- 172.21.15.12:0/3672028127 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cac02cc60 con 0x7f3cbc0d3170 2024-04-15T15:21:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc1357640 1 -- 172.21.15.12:0/3672028127 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac034660 con 0x7f3cbc0d3170 2024-04-15T15:21:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc45bd640 1 -- 172.21.15.12:0/3672028127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc0d3170 msgr2=0x7f3cbc06d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc45bd640 1 --2- 172.21.15.12:0/3672028127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc0d3170 0x7f3cbc06d220 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009d90 tx=0x7f3cac02beb0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc45bd640 1 -- 172.21.15.12:0/3672028127 shutdown_connections 2024-04-15T15:21:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc45bd640 1 --2- 172.21.15.12:0/3672028127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc0d3170 0x7f3cbc06d220 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc45bd640 1 -- 172.21.15.12:0/3672028127 >> 172.21.15.12:0/3672028127 conn(0x7f3cbc08bf60 msgr2=0x7f3cbc0d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc45bd640 1 -- 172.21.15.12:0/3672028127 shutdown_connections 2024-04-15T15:21:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.664+0000 7f3cc45bd640 1 -- 172.21.15.12:0/3672028127 wait complete. 2024-04-15T15:21:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.665+0000 7f3cc45bd640 1 Processor -- start 2024-04-15T15:21:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.665+0000 7f3cc45bd640 1 -- start start 2024-04-15T15:21:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc45bd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc0d15c0 0x7f3cbc0cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc45bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d3170 0x7f3cbc157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc45bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e7a0 0x7f3cbc0ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc45bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc15eb80 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc45bd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc0cf180 con 0x7f3cbc0d15c0 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc45bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3cbc0d1a40 con 0x7f3cbc0d3170 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc2b5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e7a0 0x7f3cbc0ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc2b5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e7a0 0x7f3cbc0ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47126/0 (socket says 172.21.15.12:47126) 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.666+0000 7f3cc2b5a640 1 -- 172.21.15.12:0/2701199590 learned_addr learned my addr 172.21.15.12:0/2701199590 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc2359640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc0d15c0 0x7f3cbc0cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc1b58640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d3170 0x7f3cbc157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc2b5a640 1 -- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d3170 msgr2=0x7f3cbc157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc2b5a640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d3170 0x7f3cbc157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc2b5a640 1 -- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc0d15c0 msgr2=0x7f3cbc0cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc2b5a640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc0d15c0 0x7f3cbc0cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc2b5a640 1 -- 172.21.15.12:0/2701199590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cbc0b8330 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc2359640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc0d15c0 0x7f3cbc0cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc2b5a640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e7a0 0x7f3cbc0ca930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009d90 tx=0x7f3cac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2701199590 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac039050 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2701199590 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cac045070 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2701199590 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac03e830 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cbc0b8610 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.667+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cbc0b8bd0 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.668+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cbc0689e0 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.668+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2701199590 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cac03e9d0 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.669+0000 7f3cbaffd640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.669+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2701199590 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3cac0aec90 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.669+0000 7f3cc2359640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.670+0000 7f3cc2359640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4000f60 tx=0x7f3cb4007c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.673+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2701199590 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cac07cd80 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.860+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3cbc0d1d60 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.860+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2701199590 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f3cac07c680 con 0x7f3cbc15e7a0 2024-04-15T15:21:41.862 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 msgr2=0x7f3ca806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4000f60 tx=0x7f3cb4007c50 comp rx=0 tx=0).stop 2024-04-15T15:21:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e7a0 msgr2=0x7f3cbc0ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e7a0 0x7f3cbc0ca930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009d90 tx=0x7f3cac004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 shutdown_connections 2024-04-15T15:21:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e7a0 0x7f3cbc0ca930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d3170 0x7f3cbc157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 --2- 172.21.15.12:0/2701199590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc0d15c0 0x7f3cbc0cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.862+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 >> 172.21.15.12:0/2701199590 conn(0x7f3cbc08bf60 msgr2=0x7f3cbc0b42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.863+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 shutdown_connections 2024-04-15T15:21:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.863+0000 7f3cc45bd640 1 -- 172.21.15.12:0/2701199590 wait complete. 2024-04-15T15:21:41.876 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:21:41.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.979+0000 7f1e1df96640 1 Processor -- start 2024-04-15T15:21:41.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.979+0000 7f1e1df96640 1 -- start start 2024-04-15T15:21:41.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.980+0000 7f1e1df96640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e180d0d60 0x7f1e1806c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.980+0000 7f1e1df96640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e1815ee00 con 0x7f1e1815ea20 2024-04-15T15:21:41.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.980+0000 7f1e1df96640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e180b24f0 con 0x7f1e180b1ae0 2024-04-15T15:21:41.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.980+0000 7f1e1df96640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1e180b22c0 con 0x7f1e180d0d60 2024-04-15T15:21:41.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.980+0000 7f1e17fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e180d0d60 0x7f1e1806c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.980+0000 7f1e177fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1e1815ea20 0x7f1e180d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56492/0 (socket says 172.21.15.12:56492) 2024-04-15T15:21:41.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.980+0000 7f1e177fe640 1 -- 172.21.15.12:0/3037211456 learned_addr learned my addr 172.21.15.12:0/3037211456 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:41.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.981+0000 7f1e17fff640 1 -- 172.21.15.12:0/3037211456 >> v1:172.21.15.55:6789/0 conn(0x7f1e180b1ae0 legacy=0x7f1e180cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:41.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.981+0000 7f1e17fff640 1 -- 172.21.15.12:0/3037211456 >> v1:172.21.15.12:6789/0 conn(0x7f1e1815ea20 legacy=0x7f1e180d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.981+0000 7f1e17fff640 1 -- 172.21.15.12:0/3037211456 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e180b27d0 con 0x7f1e180d0d60 2024-04-15T15:21:41.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.981+0000 7f1e17fff640 1 --2- 172.21.15.12:0/3037211456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e1806c3f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1e100097a0 tx=0x7f1e1002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5677c5e948e7114a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e167fc640 1 -- 172.21.15.12:0/3037211456 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e100315a0 con 0x7f1e180d0d60 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e167fc640 1 -- 172.21.15.12:0/3037211456 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e10037070 con 0x7f1e180d0d60 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e167fc640 1 -- 172.21.15.12:0/3037211456 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10031960 con 0x7f1e180d0d60 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e1df96640 1 -- 172.21.15.12:0/3037211456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 msgr2=0x7f1e1806c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e1df96640 1 --2- 172.21.15.12:0/3037211456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e1806c3f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1e100097a0 tx=0x7f1e1002b840 comp rx=0 tx=0).stop 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e1df96640 1 -- 172.21.15.12:0/3037211456 shutdown_connections 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e1df96640 1 --2- 172.21.15.12:0/3037211456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e1806c3f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e1df96640 1 -- 172.21.15.12:0/3037211456 >> 172.21.15.12:0/3037211456 conn(0x7f1e1809a9d0 msgr2=0x7f1e180b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e1df96640 1 -- 172.21.15.12:0/3037211456 shutdown_connections 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.982+0000 7f1e1df96640 1 -- 172.21.15.12:0/3037211456 wait complete. 2024-04-15T15:21:41.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.983+0000 7f1e1df96640 1 Processor -- start 2024-04-15T15:21:41.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.983+0000 7f1e1df96640 1 -- start start 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e1df96640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e180b1ae0 0x7f1e180c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e1df96640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e18156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e1df96640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1815ea20 0x7f1e18161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e1df96640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1e180b2d90 con 0x7f1e180d0d60 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e1df96640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1e180b0640 con 0x7f1e1815ea20 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e1df96640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1e180ce210 con 0x7f1e180b1ae0 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e16ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e18156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e16ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e18156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47150/0 (socket says 172.21.15.12:47150) 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e16ffd640 1 -- 172.21.15.12:0/2405915029 learned_addr learned my addr 172.21.15.12:0/2405915029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e17fff640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1815ea20 0x7f1e18161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.984+0000 7f1e177fe640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e180b1ae0 0x7f1e180c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1e16ffd640 1 -- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e180b1ae0 msgr2=0x7f1e180c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1e16ffd640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e180b1ae0 0x7f1e180c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1e16ffd640 1 -- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1815ea20 msgr2=0x7f1e18161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1e16ffd640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1815ea20 0x7f1e18161000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1e16ffd640 1 -- 172.21.15.12:0/2405915029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e180c5a50 con 0x7f1e180d0d60 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1e16ffd640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e18156f10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c460 tx=0x7f1e0c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1dfffff640 1 -- 172.21.15.12:0/2405915029 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c016020 con 0x7f1e180d0d60 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1dfffff640 1 -- 172.21.15.12:0/2405915029 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e0c012870 con 0x7f1e180d0d60 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e180c7d40 con 0x7f1e180d0d60 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1dfffff640 1 -- 172.21.15.12:0/2405915029 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c012c60 con 0x7f1e180d0d60 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.985+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e180c5fb0 con 0x7f1e180d0d60 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.986+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e180690f0 con 0x7f1e180d0d60 2024-04-15T15:21:41.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.987+0000 7f1dfffff640 1 -- 172.21.15.12:0/2405915029 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e0c019020 con 0x7f1e180d0d60 2024-04-15T15:21:41.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.987+0000 7f1dfffff640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e0406c8f0 0x7f1e0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:41.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.987+0000 7f1dfffff640 1 -- 172.21.15.12:0/2405915029 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1e0c08b9c0 con 0x7f1e180d0d60 2024-04-15T15:21:41.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.987+0000 7f1e177fe640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e0406c8f0 0x7f1e0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:41.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.988+0000 7f1e177fe640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e0406c8f0 0x7f1e0406edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1e080012e0 tx=0x7f1e08001990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:41.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:41.991+0000 7f1dfffff640 1 -- 172.21.15.12:0/2405915029 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e0c08de20 con 0x7f1e180d0d60 2024-04-15T15:21:42.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.215+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e180cc0a0 con 0x7f1e180d0d60 2024-04-15T15:21:42.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.215+0000 7f1dfffff640 1 -- 172.21.15.12:0/2405915029 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f1e0c020030 con 0x7f1e180d0d60 2024-04-15T15:21:42.217 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:42.217 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:33.113220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:21:42.217 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-15T15:21:42.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e0406c8f0 msgr2=0x7f1e0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e0406c8f0 0x7f1e0406edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1e080012e0 tx=0x7f1e08001990 comp rx=0 tx=0).stop 2024-04-15T15:21:42.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 msgr2=0x7f1e18156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e18156f10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c460 tx=0x7f1e0c00c920 comp rx=0 tx=0).stop 2024-04-15T15:21:42.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 shutdown_connections 2024-04-15T15:21:42.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e0406c8f0 0x7f1e0406edb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1815ea20 0x7f1e18161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e180d0d60 0x7f1e18156f10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 --2- 172.21.15.12:0/2405915029 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e180b1ae0 0x7f1e180c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 >> 172.21.15.12:0/2405915029 conn(0x7f1e1809a9d0 msgr2=0x7f1e1806c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:42.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 shutdown_connections 2024-04-15T15:21:42.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.217+0000 7f1e1df96640 1 -- 172.21.15.12:0/2405915029 wait complete. 2024-04-15T15:21:42.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.333+0000 7faab253d640 1 Processor -- start 2024-04-15T15:21:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.333+0000 7faab253d640 1 -- start start 2024-04-15T15:21:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.334+0000 7faab253d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7faaac15e7a0 0x7faaac0d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.334+0000 7faab253d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaac0d0530 con 0x7faaac0d13f0 2024-04-15T15:21:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.334+0000 7faab253d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaac163190 con 0x7faaac0d1870 2024-04-15T15:21:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.334+0000 7faab253d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7faaac0cf490 con 0x7faaac15e7a0 2024-04-15T15:21:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.334+0000 7faaab7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7faaac0d13f0 0x7faaac162370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44396/0 (socket says 172.21.15.12:44396) 2024-04-15T15:21:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.334+0000 7faaab7fe640 1 -- 172.21.15.12:0/3628954203 learned_addr learned my addr 172.21.15.12:0/3628954203 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.334+0000 7faaabfff640 1 --2- 172.21.15.12:0/3628954203 >> v2:172.21.15.55:3300/0 conn(0x7faaac15e7a0 0x7faaac0d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.335+0000 7faaaaffd640 1 -- 172.21.15.12:0/3628954203 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1328918611 0 0) 0x7faaac0d0530 con 0x7faaac0d13f0 2024-04-15T15:21:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.335+0000 7faaaaffd640 1 -- 172.21.15.12:0/3628954203 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa94003520 con 0x7faaac0d13f0 2024-04-15T15:21:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.335+0000 7faaabfff640 1 -- 172.21.15.12:0/3628954203 >> v1:172.21.15.55:6789/0 conn(0x7faaac0d1870 legacy=0x7faaac162a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.335+0000 7faaabfff640 1 -- 172.21.15.12:0/3628954203 >> v1:172.21.15.12:6789/0 conn(0x7faaac0d13f0 legacy=0x7faaac162370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.335+0000 7faaabfff640 1 -- 172.21.15.12:0/3628954203 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaac0cf770 con 0x7faaac15e7a0 2024-04-15T15:21:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faaabfff640 1 --2- 172.21.15.12:0/3628954203 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac15e7a0 0x7faaac0d0eb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0097a0 tx=0x7faa9c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5184f4069980328 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faaaaffd640 1 -- 172.21.15.12:0/3628954203 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c035020 con 0x7faaac15e7a0 2024-04-15T15:21:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faaaaffd640 1 -- 172.21.15.12:0/3628954203 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa9c031830 con 0x7faaac15e7a0 2024-04-15T15:21:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faaaaffd640 1 -- 172.21.15.12:0/3628954203 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c031be0 con 0x7faaac15e7a0 2024-04-15T15:21:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faab253d640 1 -- 172.21.15.12:0/3628954203 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac15e7a0 msgr2=0x7faaac0d0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faab253d640 1 --2- 172.21.15.12:0/3628954203 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac15e7a0 0x7faaac0d0eb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0097a0 tx=0x7faa9c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faab253d640 1 -- 172.21.15.12:0/3628954203 shutdown_connections 2024-04-15T15:21:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faab253d640 1 --2- 172.21.15.12:0/3628954203 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac15e7a0 0x7faaac0d0eb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faab253d640 1 -- 172.21.15.12:0/3628954203 >> 172.21.15.12:0/3628954203 conn(0x7faaac08bf60 msgr2=0x7faaac0d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faab253d640 1 -- 172.21.15.12:0/3628954203 shutdown_connections 2024-04-15T15:21:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.336+0000 7faab253d640 1 -- 172.21.15.12:0/3628954203 wait complete. 2024-04-15T15:21:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.337+0000 7faab253d640 1 Processor -- start 2024-04-15T15:21:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.338+0000 7faab253d640 1 -- start start 2024-04-15T15:21:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.338+0000 7faab253d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faaac0d13f0 0x7faaac09e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.338+0000 7faab253d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac0d1870 0x7faaac14e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.338+0000 7faab253d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaac15e7a0 0x7faaac14b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.338+0000 7faab253d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faaac0cf180 con 0x7faaac15e7a0 2024-04-15T15:21:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.338+0000 7faab253d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faaac0cfd30 con 0x7faaac0d1870 2024-04-15T15:21:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.338+0000 7faab253d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faaac06db20 con 0x7faaac0d13f0 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaac15e7a0 0x7faaac14b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaac15e7a0 0x7faaac14b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47744/0 (socket says 172.21.15.12:47744) 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 -- 172.21.15.12:0/3268511022 learned_addr learned my addr 172.21.15.12:0/3268511022 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faaab7fe640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac0d1870 0x7faaac14e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faaabfff640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faaac0d13f0 0x7faaac09e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 -- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faaac0d13f0 msgr2=0x7faaac09e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faaac0d13f0 0x7faaac09e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 -- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac0d1870 msgr2=0x7faaac14e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac0d1870 0x7faaac14e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 -- 172.21.15.12:0/3268511022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaac0ca720 con 0x7faaac15e7a0 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab0ada640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaac15e7a0 0x7faaac14b0b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c7a0 tx=0x7faaa400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faaa8ff9640 1 -- 172.21.15.12:0/3268511022 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4003b70 con 0x7faaac15e7a0 2024-04-15T15:21:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faaa8ff9640 1 -- 172.21.15.12:0/3268511022 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa4003d30 con 0x7faaac15e7a0 2024-04-15T15:21:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaac14eb50 con 0x7faaac15e7a0 2024-04-15T15:21:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faaa8ff9640 1 -- 172.21.15.12:0/3268511022 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa400aa30 con 0x7faaac15e7a0 2024-04-15T15:21:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.339+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaac14de10 con 0x7faaac15e7a0 2024-04-15T15:21:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.340+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaac0694e0 con 0x7faaac15e7a0 2024-04-15T15:21:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.341+0000 7faaa8ff9640 1 -- 172.21.15.12:0/3268511022 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faaa400abd0 con 0x7faaac15e7a0 2024-04-15T15:21:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.341+0000 7faaa8ff9640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa9806c8f0 0x7faa9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.342+0000 7faaa8ff9640 1 -- 172.21.15.12:0/3268511022 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faaa408be10 con 0x7faaac15e7a0 2024-04-15T15:21:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.342+0000 7faaabfff640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa9806c8f0 0x7faa9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.342+0000 7faaabfff640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa9806c8f0 0x7faa9806edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faa9c036000 tx=0x7faa9c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.346+0000 7faaa8ff9640 1 -- 172.21.15.12:0/3268511022 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faaac0694e0 con 0x7faaac15e7a0 2024-04-15T15:21:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.532+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faaac0689e0 con 0x7faaac15e7a0 2024-04-15T15:21:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.533+0000 7faaa8ff9640 1 -- 172.21.15.12:0/3268511022 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7faaa408f030 con 0x7faaac15e7a0 2024-04-15T15:21:42.535 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:42.535 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:16.063979+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa9806c8f0 msgr2=0x7faa9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa9806c8f0 0x7faa9806edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faa9c036000 tx=0x7faa9c008030 comp rx=0 tx=0).stop 2024-04-15T15:21:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaac15e7a0 msgr2=0x7faaac14b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaac15e7a0 0x7faaac14b0b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c7a0 tx=0x7faaa400cc60 comp rx=0 tx=0).stop 2024-04-15T15:21:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 shutdown_connections 2024-04-15T15:21:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaac15e7a0 0x7faaac14b0b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa9806c8f0 0x7faa9806edb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faaac0d1870 0x7faaac14e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 --2- 172.21.15.12:0/3268511022 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faaac0d13f0 0x7faaac09e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 >> 172.21.15.12:0/3268511022 conn(0x7faaac08bf60 msgr2=0x7faaac0b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 shutdown_connections 2024-04-15T15:21:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.535+0000 7faab253d640 1 -- 172.21.15.12:0/3268511022 wait complete. 2024-04-15T15:21:42.548 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:21:42.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.653+0000 7f990d2e4640 1 Processor -- start 2024-04-15T15:21:42.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.653+0000 7f990d2e4640 1 -- start start 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.654+0000 7f990d2e4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99080b0850 0x7f99080705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.654+0000 7f990d2e4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080aeed0 0x7f99080af360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.654+0000 7f990d2e4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990815ebc0 con 0x7f990806b9e0 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.654+0000 7f990d2e4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f99080ae150 con 0x7f99080b0850 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.654+0000 7f990d2e4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9908070c80 con 0x7f99080aeed0 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.654+0000 7f99067fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99080b0850 0x7f99080705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.654+0000 7f99067fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f99080b0850 0x7f99080705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47748/0 (socket says 172.21.15.12:47748) 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.654+0000 7f99067fc640 1 -- 172.21.15.12:0/3673725690 learned_addr learned my addr 172.21.15.12:0/3673725690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.655+0000 7f99077fe640 1 --2- 172.21.15.12:0/3673725690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080aeed0 0x7f99080af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.655+0000 7f99067fc640 1 -- 172.21.15.12:0/3673725690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080aeed0 msgr2=0x7f99080af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.655+0000 7f99067fc640 1 --2- 172.21.15.12:0/3673725690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080aeed0 0x7f99080af360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.655+0000 7f99067fc640 1 -- 172.21.15.12:0/3673725690 >> v1:172.21.15.55:6789/0 conn(0x7f990806b9e0 legacy=0x7f990806bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.655+0000 7f99067fc640 1 -- 172.21.15.12:0/3673725690 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99080ae3f0 con 0x7f99080b0850 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.655+0000 7f99077fe640 1 --2- 172.21.15.12:0/3673725690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080aeed0 0x7f99080af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.655+0000 7f99067fc640 1 --2- 172.21.15.12:0/3673725690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99080b0850 0x7f99080705a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98f00098a0 tx=0x7f98f002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5fe84e2b995086e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f9905ffb640 1 -- 172.21.15.12:0/3673725690 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0034070 con 0x7f99080b0850 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f9905ffb640 1 -- 172.21.15.12:0/3673725690 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f0030840 con 0x7f99080b0850 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f9905ffb640 1 -- 172.21.15.12:0/3673725690 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0030c10 con 0x7f99080b0850 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f990d2e4640 1 -- 172.21.15.12:0/3673725690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99080b0850 msgr2=0x7f99080705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f990d2e4640 1 --2- 172.21.15.12:0/3673725690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99080b0850 0x7f99080705a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98f00098a0 tx=0x7f98f002b980 comp rx=0 tx=0).stop 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f990d2e4640 1 -- 172.21.15.12:0/3673725690 shutdown_connections 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f990d2e4640 1 --2- 172.21.15.12:0/3673725690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99080b0850 0x7f99080705a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f990d2e4640 1 --2- 172.21.15.12:0/3673725690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080aeed0 0x7f99080af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f990d2e4640 1 -- 172.21.15.12:0/3673725690 >> 172.21.15.12:0/3673725690 conn(0x7f99080b2e70 msgr2=0x7f99080b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:42.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f990d2e4640 1 -- 172.21.15.12:0/3673725690 shutdown_connections 2024-04-15T15:21:42.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.656+0000 7f990d2e4640 1 -- 172.21.15.12:0/3673725690 wait complete. 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f990d2e4640 1 Processor -- start 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f990d2e4640 1 -- start start 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f990d2e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990806b9e0 0x7f990814ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f990d2e4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99080aeed0 0x7f99080b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f990d2e4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080b0850 0x7f99081616d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f990d2e4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f99080cf0a0 con 0x7f990806b9e0 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f990d2e4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f99080ae9b0 con 0x7f99080aeed0 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f990d2e4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f99080af8a0 con 0x7f99080b0850 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f9906ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990806b9e0 0x7f990814ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f9906ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990806b9e0 0x7f990814ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47762/0 (socket says 172.21.15.12:47762) 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.658+0000 7f9906ffd640 1 -- 172.21.15.12:0/723620843 learned_addr learned my addr 172.21.15.12:0/723620843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f99067fc640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99080aeed0 0x7f99080b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f99077fe640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080b0850 0x7f99081616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f9906ffd640 1 -- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080b0850 msgr2=0x7f99081616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f9906ffd640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080b0850 0x7f99081616d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f9906ffd640 1 -- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99080aeed0 msgr2=0x7f99080b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f9906ffd640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99080aeed0 0x7f99080b3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f9906ffd640 1 -- 172.21.15.12:0/723620843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990814fc90 con 0x7f990806b9e0 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f99077fe640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080b0850 0x7f99081616d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f99067fc640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99080aeed0 0x7f99080b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f9906ffd640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990806b9e0 0x7f990814ca10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f990000c430 tx=0x7f990000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f98ff7fe640 1 -- 172.21.15.12:0/723620843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000052d0 con 0x7f990806b9e0 2024-04-15T15:21:42.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f98ff7fe640 1 -- 172.21.15.12:0/723620843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9900005490 con 0x7f990806b9e0 2024-04-15T15:21:42.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99080b6760 con 0x7f990806b9e0 2024-04-15T15:21:42.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f98ff7fe640 1 -- 172.21.15.12:0/723620843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900007260 con 0x7f990806b9e0 2024-04-15T15:21:42.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.659+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99080b6d20 con 0x7f990806b9e0 2024-04-15T15:21:42.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.660+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99080690f0 con 0x7f990806b9e0 2024-04-15T15:21:42.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.661+0000 7f98ff7fe640 1 -- 172.21.15.12:0/723620843 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9900007400 con 0x7f990806b9e0 2024-04-15T15:21:42.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.661+0000 7f98ff7fe640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f98e806c8f0 0x7f98e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:42.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.661+0000 7f98ff7fe640 1 -- 172.21.15.12:0/723620843 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f990008b360 con 0x7f990806b9e0 2024-04-15T15:21:42.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.661+0000 7f99067fc640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f98e806c8f0 0x7f98e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:42.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.662+0000 7f99067fc640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f98e806c8f0 0x7f98e806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f98f0036000 tx=0x7f98f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:42.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.666+0000 7f98ff7fe640 1 -- 172.21.15.12:0/723620843 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99000594e0 con 0x7f990806b9e0 2024-04-15T15:21:42.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.895+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f990815df50 con 0x7f990806b9e0 2024-04-15T15:21:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.895+0000 7f98ff7fe640 1 -- 172.21.15.12:0/723620843 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f9900058de0 con 0x7f990806b9e0 2024-04-15T15:21:42.897 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:42.897 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:33.113220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:21:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f98e806c8f0 msgr2=0x7f98e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f98e806c8f0 0x7f98e806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f98f0036000 tx=0x7f98f0008030 comp rx=0 tx=0).stop 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990806b9e0 msgr2=0x7f990814ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990806b9e0 0x7f990814ca10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f990000c430 tx=0x7f990000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 shutdown_connections 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f99080b0850 0x7f99081616d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f98e806c8f0 0x7f98e806edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99080aeed0 0x7f99080b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 --2- 172.21.15.12:0/723620843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f990806b9e0 0x7f990814ca10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 >> 172.21.15.12:0/723620843 conn(0x7f99080b2e70 msgr2=0x7f9908074ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 shutdown_connections 2024-04-15T15:21:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:42.897+0000 7f990d2e4640 1 -- 172.21.15.12:0/723620843 wait complete. 2024-04-15T15:21:43.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:21:11.000401+0000', 'modified': '2024-04-15T15:21:33.113220+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.55:6837/1379664958', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 1379664958}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 1379664958}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4290, 'qdb_cluster': [4290]} 2024-04-15T15:21:43.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:21:43.313 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-15T15:21:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.419+0000 7fc211eba640 1 Processor -- start 2024-04-15T15:21:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.419+0000 7fc211eba640 1 -- start start 2024-04-15T15:21:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc211eba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c0ce8e0 0x7fc20c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc211eba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc20c071d90 0x7fc20c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc211eba640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc20c071240 0x7fc20c09af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc211eba640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc20c070820 con 0x7fc20c071240 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc211eba640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc20c06ff30 con 0x7fc20c071d90 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc211eba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc20c0cecc0 con 0x7fc20c0ce8e0 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc20c071240 0x7fc20c09af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc20c071240 0x7fc20c09af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47766/0 (socket says 172.21.15.12:47766) 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20bfff640 1 -- 172.21.15.12:0/2398479015 learned_addr learned my addr 172.21.15.12:0/2398479015 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/2398479015 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c0ce8e0 0x7fc20c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20affd640 1 --2- 172.21.15.12:0/2398479015 >> v2:172.21.15.55:3300/0 conn(0x7fc20c071d90 0x7fc20c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20bfff640 1 -- 172.21.15.12:0/2398479015 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c0ce8e0 msgr2=0x7fc20c070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20bfff640 1 --2- 172.21.15.12:0/2398479015 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c0ce8e0 0x7fc20c070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20bfff640 1 -- 172.21.15.12:0/2398479015 >> v2:172.21.15.55:3300/0 conn(0x7fc20c071d90 msgr2=0x7fc20c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.420+0000 7fc20bfff640 1 --2- 172.21.15.12:0/2398479015 >> v2:172.21.15.55:3300/0 conn(0x7fc20c071d90 0x7fc20c09a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.421+0000 7fc20bfff640 1 -- 172.21.15.12:0/2398479015 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20c09b9d0 con 0x7fc20c071240 2024-04-15T15:21:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.421+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/2398479015 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c0ce8e0 0x7fc20c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.421+0000 7fc20bfff640 1 --2- 172.21.15.12:0/2398479015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071240 0x7fc20c09af10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0696a0 tx=0x7fc20402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c250d6e98960d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.421+0000 7fc20a7fc640 1 -- 172.21.15.12:0/2398479015 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20402c9e0 con 0x7fc20c071240 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.421+0000 7fc20a7fc640 1 -- 172.21.15.12:0/2398479015 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20402cb80 con 0x7fc20c071240 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc20a7fc640 1 -- 172.21.15.12:0/2398479015 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2040343f0 con 0x7fc20c071240 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 -- 172.21.15.12:0/2398479015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071240 msgr2=0x7fc20c09af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 --2- 172.21.15.12:0/2398479015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071240 0x7fc20c09af10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0696a0 tx=0x7fc20402be40 comp rx=0 tx=0).stop 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 -- 172.21.15.12:0/2398479015 shutdown_connections 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 --2- 172.21.15.12:0/2398479015 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071240 0x7fc20c09af10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 --2- 172.21.15.12:0/2398479015 >> v2:172.21.15.55:3300/0 conn(0x7fc20c071d90 0x7fc20c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 --2- 172.21.15.12:0/2398479015 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c0ce8e0 0x7fc20c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 -- 172.21.15.12:0/2398479015 >> 172.21.15.12:0/2398479015 conn(0x7fc20c0b3f40 msgr2=0x7fc20c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 -- 172.21.15.12:0/2398479015 shutdown_connections 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.422+0000 7fc211eba640 1 -- 172.21.15.12:0/2398479015 wait complete. 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.423+0000 7fc211eba640 1 Processor -- start 2024-04-15T15:21:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.423+0000 7fc211eba640 1 -- start start 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc211eba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c071240 0x7fc20c152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc211eba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071d90 0x7fc20c150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc211eba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc20c0ce8e0 0x7fc20c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc211eba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c0704b0 con 0x7fc20c071d90 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc211eba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c0701d0 con 0x7fc20c0ce8e0 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc211eba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc20c071970 con 0x7fc20c071240 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc20affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071d90 0x7fc20c150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc20affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071d90 0x7fc20c150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47778/0 (socket says 172.21.15.12:47778) 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc20affd640 1 -- 172.21.15.12:0/2997651573 learned_addr learned my addr 172.21.15.12:0/2997651573 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c071240 0x7fc20c152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.424+0000 7fc20bfff640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc20c0ce8e0 0x7fc20c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc20affd640 1 -- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c071240 msgr2=0x7fc20c152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc20affd640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c071240 0x7fc20c152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc20affd640 1 -- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc20c0ce8e0 msgr2=0x7fc20c1528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc20affd640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc20c0ce8e0 0x7fc20c1528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc20affd640 1 -- 172.21.15.12:0/2997651573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20c1511e0 con 0x7fc20c071d90 2024-04-15T15:21:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc20affd640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071d90 0x7fc20c150860 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc20000e8a0 tx=0x7fc20000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc1f3fff640 1 -- 172.21.15.12:0/2997651573 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200017070 con 0x7fc20c071d90 2024-04-15T15:21:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc1f3fff640 1 -- 172.21.15.12:0/2997651573 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20000ee90 con 0x7fc20c071d90 2024-04-15T15:21:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc1f3fff640 1 -- 172.21.15.12:0/2997651573 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20000cb80 con 0x7fc20c071d90 2024-04-15T15:21:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc211eba640 1 -- 172.21.15.12:0/2997651573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc20c1514c0 con 0x7fc20c071d90 2024-04-15T15:21:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.425+0000 7fc211eba640 1 -- 172.21.15.12:0/2997651573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc20c151a80 con 0x7fc20c071d90 2024-04-15T15:21:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.426+0000 7fc1f3fff640 1 -- 172.21.15.12:0/2997651573 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc20001a070 con 0x7fc20c071d90 2024-04-15T15:21:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.427+0000 7fc1f3fff640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1f806c920 0x7fc1f806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.427+0000 7fc1f3fff640 1 -- 172.21.15.12:0/2997651573 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc20008bf80 con 0x7fc20c071d90 2024-04-15T15:21:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.427+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1f806c920 0x7fc1f806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.428+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1f806c920 0x7fc1f806ede0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0b2320 tx=0x7fc1fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.428+0000 7fc214b77740 1 Processor -- start 2024-04-15T15:21:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.428+0000 7fc214b77740 1 -- start start 2024-04-15T15:21:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc214b77740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bfe10 0x55b3d9292db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc214b77740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92c01f0 0x55b3d92c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc214b77740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92932f0 0x55b3d92c30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc214b77740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b3d91fd9c0 con 0x55b3d92c01f0 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc214b77740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b3d91a4650 con 0x55b3d92bfe10 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc214b77740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55b3d91f9980 con 0x55b3d92932f0 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92c01f0 0x55b3d92c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92c01f0 0x55b3d92c2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47782/0 (socket says 172.21.15.12:47782) 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20affd640 1 -- 172.21.15.12:0/3430254918 learned_addr learned my addr 172.21.15.12:0/3430254918 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bfe10 0x55b3d9292db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20bfff640 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92932f0 0x55b3d92c30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20affd640 1 -- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92932f0 msgr2=0x55b3d92c30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20affd640 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92932f0 0x55b3d92c30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20affd640 1 -- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bfe10 msgr2=0x55b3d9292db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20affd640 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bfe10 0x55b3d9292db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20affd640 1 -- 172.21.15.12:0/3430254918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d8f102d0 con 0x55b3d92c01f0 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.429+0000 7fc20bfff640 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92932f0 0x55b3d92c30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc20affd640 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92c01f0 0x55b3d92c2ba0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc20008a0d0 tx=0x7fc200059ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d5771121a471b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc1f17fa640 1 -- 172.21.15.12:0/3430254918 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200021070 con 0x55b3d92c01f0 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc1f17fa640 1 -- 172.21.15.12:0/3430254918 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc200057d10 con 0x55b3d92c01f0 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc1f17fa640 1 -- 172.21.15.12:0/3430254918 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200056080 con 0x55b3d92c01f0 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 -- 172.21.15.12:0/3430254918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92c01f0 msgr2=0x55b3d92c2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92c01f0 0x55b3d92c2ba0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc20008a0d0 tx=0x7fc200059ad0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 -- 172.21.15.12:0/3430254918 shutdown_connections 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92932f0 0x55b3d92c30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92c01f0 0x55b3d92c2ba0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 --2- 172.21.15.12:0/3430254918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bfe10 0x55b3d9292db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 -- 172.21.15.12:0/3430254918 >> 172.21.15.12:0/3430254918 conn(0x55b3d9295020 msgr2=0x55b3d92ddba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 -- 172.21.15.12:0/3430254918 shutdown_connections 2024-04-15T15:21:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.430+0000 7fc214b77740 1 -- 172.21.15.12:0/3430254918 wait complete. 2024-04-15T15:21:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.431+0000 7fc214b77740 1 -- start start 2024-04-15T15:21:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.431+0000 7fc214b77740 1 Processor -- start 2024-04-15T15:21:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc214b77740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc214b77740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92c01f0 0x55b3d92de3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc214b77740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bef80 0x55b3d92eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc214b77740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b3d92975c0 con 0x55b3d92bfe10 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc214b77740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b3d91b7a10 con 0x55b3d92bef80 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc214b77740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55b3d92969b0 con 0x55b3d92c01f0 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc214b77740 10 client.0 _ll_register_callbacks cb 0x55b3d91f3250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47784/0 (socket says 172.21.15.12:47784) 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 -- 172.21.15.12:0/2083991279 learned_addr learned my addr 172.21.15.12:0/2083991279 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20affd640 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92c01f0 0x55b3d92de3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20bfff640 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bef80 0x55b3d92eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 -- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92c01f0 msgr2=0x55b3d92de3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92c01f0 0x55b3d92de3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 -- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bef80 msgr2=0x55b3d92eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bef80 0x55b3d92eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 -- 172.21.15.12:0/2083991279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3d8fda6b0 con 0x55b3d92bfe10 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc016730 tx=0x7fc1fc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a08be8c600a4ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.432+0000 7fc1ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/2083991279 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc01c070 con 0x55b3d92bfe10 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/2083991279 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc1fc0147b0 con 0x55b3d92bfe10 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 1 -- 172.21.15.12:0/2083991279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc200057d10 con 0x55b3d92bfe10 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/2083991279 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7fc1fc02f050 con 0x55b3d92bfe10 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 10 client.4821 fetch_fsmap learned FSMap version 9 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 1 -- 172.21.15.12:0/2083991279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x55b3d92ee850 con 0x55b3d92bfe10 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/2083991279 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc200057d10 con 0x55b3d92bfe10 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/2083991279 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7fc1fc032020 con 0x55b3d92bfe10 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 10 client.4821 fetch_fsmap finished waiting for FSMap version 9 2024-04-15T15:21:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 10 client.4821 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 20 client.4821 populate_metadata read hostname 'smithi012' 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55b3d92d5370 0x55b3d92fc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 4 client.4821 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc214b77740 1 -- 172.21.15.12:0/2083991279 --> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3d920b3e0 con 0x55b3d92d5370 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.433+0000 7fc20affd640 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55b3d92d5370 0x55b3d92fc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.434+0000 7fc1ea7fc640 10 client.4821 ms_handle_connect on v2:172.21.15.55:6836/1379664958 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.434+0000 7fc20affd640 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55b3d92d5370 0x55b3d92fc6e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73bd4c19d2ab0c7b server_cookie=194be9aba884eb52 in_seq=0 out_seq=0 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.434+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/2083991279 <== mds.0 v2:172.21.15.55:6836/1379664958 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3d920b3e0 con 0x55b3d92d5370 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.434+0000 7fc1ea7fc640 10 client.4821 handle_command_reply: tid=0 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.434+0000 7fc214b77740 1 client.4821 shutdown 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 -- 172.21.15.12:0/2083991279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 msgr2=0x55b3d92edc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc016730 tx=0x7fc1fc013000 comp rx=0 tx=0).stop 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 -- 172.21.15.12:0/2083991279 shutdown_connections 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55b3d92d5370 0x55b3d92fc6e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92bef80 0x55b3d92eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc1ea7fc640 0 client.4821 ms_handle_reset on v2:172.21.15.55:6836/1379664958 2024-04-15T15:21:43.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92c01f0 0x55b3d92de3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 --2- 172.21.15.12:0/2083991279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 -- 172.21.15.12:0/2083991279 >> 172.21.15.12:0/2083991279 conn(0x55b3d92932f0 msgr2=0x55b3d92e10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:43.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 -- 172.21.15.12:0/2083991279 shutdown_connections 2024-04-15T15:21:43.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 1 -- 172.21.15.12:0/2083991279 wait complete. 2024-04-15T15:21:43.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.435+0000 7fc214b77740 20 client.4821 trim_cache size 0 max 16384 2024-04-15T15:21:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.453+0000 7fc214b77740 1 Processor -- start 2024-04-15T15:21:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.453+0000 7fc214b77740 1 -- start start 2024-04-15T15:21:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc214b77740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 0x55b3d92d7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc214b77740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc214b77740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d928e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc214b77740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c201330 con 0x55b3d92bfe10 2024-04-15T15:21:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc214b77740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c09eaa0 con 0x55b3d92c01f0 2024-04-15T15:21:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc214b77740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc20c200d10 con 0x55b3d92bef80 2024-04-15T15:21:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47800/0 (socket says 172.21.15.12:47800) 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 -- 172.21.15.12:0/1188886509 learned_addr learned my addr 172.21.15.12:0/1188886509 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20bfff640 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d928e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 0x55b3d92d7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 -- 172.21.15.12:0/1188886509 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 msgr2=0x55b3d92d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 0x55b3d92d7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 -- 172.21.15.12:0/1188886509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 msgr2=0x55b3d928e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d928e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 -- 172.21.15.12:0/1188886509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d928eda0 con 0x55b3d92bfe10 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc20affd640 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc20000e870 tx=0x7fc200021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748f4d9a86243df7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.454+0000 7fc1f0ff9640 1 -- 172.21.15.12:0/1188886509 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200057d10 con 0x55b3d92bfe10 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc1f0ff9640 1 -- 172.21.15.12:0/1188886509 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20000cd20 con 0x55b3d92bfe10 2024-04-15T15:21:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc1f0ff9640 1 -- 172.21.15.12:0/1188886509 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2000600c0 con 0x55b3d92bfe10 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 -- 172.21.15.12:0/1188886509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 msgr2=0x55b3d92edc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc20000e870 tx=0x7fc200021000 comp rx=0 tx=0).stop 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 -- 172.21.15.12:0/1188886509 shutdown_connections 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d928e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d92edc00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 --2- 172.21.15.12:0/1188886509 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 0x55b3d92d7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 -- 172.21.15.12:0/1188886509 >> 172.21.15.12:0/1188886509 conn(0x55b3d92932f0 msgr2=0x55b3d92be460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 -- 172.21.15.12:0/1188886509 shutdown_connections 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.455+0000 7fc214b77740 1 -- 172.21.15.12:0/1188886509 wait complete. 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.456+0000 7fc214b77740 1 -- start start 2024-04-15T15:21:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.456+0000 7fc214b77740 1 Processor -- start 2024-04-15T15:21:43.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 0x55b3d9340fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d9300700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d92dd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b3d92ef310 con 0x55b3d92bfe10 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b3d92f1290 con 0x55b3d92c01f0 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55b3d92c5650 con 0x55b3d92bef80 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 10 client.0 _ll_register_callbacks cb 0x7fc1fc004280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d9300700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d9300700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47810/0 (socket says 172.21.15.12:47810) 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 -- 172.21.15.12:0/231496079 learned_addr learned my addr 172.21.15.12:0/231496079 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 0x55b3d9340fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20bfff640 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d92dd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 -- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 msgr2=0x55b3d9340fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 0x55b3d9340fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 -- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 msgr2=0x55b3d92dd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d92dd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 -- 172.21.15.12:0/231496079 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3d92f0d50 con 0x55b3d92bfe10 2024-04-15T15:21:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20bfff640 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d92dd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc20affd640 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d9300700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc200059b10 tx=0x7fc200009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3feb8cde07054e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc1ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/231496079 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200021040 con 0x55b3d92bfe10 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/231496079 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc200056080 con 0x55b3d92bfe10 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 1 -- 172.21.15.12:0/231496079 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc20000cd20 con 0x55b3d92bfe10 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/231496079 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7fc2000600c0 con 0x55b3d92bfe10 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 10 client.4827 fetch_fsmap learned FSMap version 9 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.457+0000 7fc214b77740 1 -- 172.21.15.12:0/231496079 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x55b3d92c55c0 con 0x55b3d92bfe10 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/231496079 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20000cd20 con 0x55b3d92bfe10 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/231496079 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7fc200058e60 con 0x55b3d92bfe10 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc214b77740 10 client.4827 fetch_fsmap finished waiting for FSMap version 9 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc214b77740 10 client.4827 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc214b77740 20 client.4827 populate_metadata read hostname 'smithi012' 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc214b77740 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55b3d92d5370 0x55b3d9296970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc214b77740 4 client.4827 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc214b77740 1 -- 172.21.15.12:0/231496079 --> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] -- command(tid 0: {"prefix": "status"}) -- 0x55b3d9205250 con 0x55b3d92d5370 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55b3d92d5370 0x55b3d9296970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc1ea7fc640 10 client.4827 ms_handle_connect on v2:172.21.15.55:6836/1379664958 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.458+0000 7fc20b7fe640 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55b3d92d5370 0x55b3d9296970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a718f3ae13826d server_cookie=bfc5b5251adb5f3e in_seq=0 out_seq=0 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc1ea7fc640 1 -- 172.21.15.12:0/231496079 <== mds.0 v2:172.21.15.55:6836/1379664958 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55b3d9205250 con 0x55b3d92d5370 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc1ea7fc640 10 client.4827 handle_command_reply: tid=0 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 client.4827 shutdown 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 -- 172.21.15.12:0/231496079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 msgr2=0x55b3d9300700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d9300700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc200059b10 tx=0x7fc200009530 comp rx=0 tx=0).stop 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 -- 172.21.15.12:0/231496079 shutdown_connections 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:6836/1379664958,v1:172.21.15.55:6837/1379664958] conn(0x55b3d92d5370 0x55b3d9296970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b3d92c01f0 0x55b3d92dd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3d92bfe10 0x55b3d9300700 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc1ea7fc640 0 client.4827 ms_handle_reset on v2:172.21.15.55:6836/1379664958 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 --2- 172.21.15.12:0/231496079 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b3d92bef80 0x55b3d9340fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 -- 172.21.15.12:0/231496079 >> 172.21.15.12:0/231496079 conn(0x55b3d92932f0 msgr2=0x55b3d92d7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 -- 172.21.15.12:0/231496079 shutdown_connections 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 1 -- 172.21.15.12:0/231496079 wait complete. 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc214b77740 20 client.4827 trim_cache size 0 max 16384 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "928fa63b-2322-46eb-a6b8-ef2b880cd7c9", 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-04-15T15:21:43.467 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4290, 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 31.832453478000001, 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 9, 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 25, 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 25, 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 39.285869601000002 2024-04-15T15:21:43.468 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 -- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1f806c920 msgr2=0x7fc1f806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1f806c920 0x7fc1f806ede0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0b2320 tx=0x7fc1fc008040 comp rx=0 tx=0).stop 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 -- 172.21.15.12:0/2997651573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071d90 msgr2=0x7fc20c150860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071d90 0x7fc20c150860 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc20000e8a0 tx=0x7fc20000ed60 comp rx=0 tx=0).stop 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 -- 172.21.15.12:0/2997651573 shutdown_connections 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1f806c920 0x7fc1f806ede0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc20c0ce8e0 0x7fc20c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc20c071d90 0x7fc20c150860 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 --2- 172.21.15.12:0/2997651573 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc20c071240 0x7fc20c152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.459+0000 7fc211eba640 1 -- 172.21.15.12:0/2997651573 >> 172.21.15.12:0/2997651573 conn(0x7fc20c0b3f40 msgr2=0x7fc20c15f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:43.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.460+0000 7fc211eba640 1 -- 172.21.15.12:0/2997651573 shutdown_connections 2024-04-15T15:21:43.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.460+0000 7fc211eba640 1 -- 172.21.15.12:0/2997651573 wait complete. 2024-04-15T15:21:43.472 INFO:teuthology.run_tasks:Running task sequential... 2024-04-15T15:21:43.481 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-15T15:21:43.481 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-15T15:21:43.481 INFO:teuthology.task.print:Enabling mgr modules 2024-04-15T15:21:43.481 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-15T15:21:43.520 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-15T15:21:43.520 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-15T15:21:43.520 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-15T15:21:43.521 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:21:43.521 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:43.521 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-15T15:21:43.521 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:21:43.521 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:21:43.521 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-15T15:21:43.521 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:21:43.521 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:43.521 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-15T15:21:43.521 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:21:43.521 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:21:43.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:43.522 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-15T15:21:43.537 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-15T15:21:43.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:43.538 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:21:43.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:43.605 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-04-15T15:21:43.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:43.711 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-04-15T15:21:43.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:43.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link delete ceph-brx 2024-04-15T15:21:43.804 INFO:teuthology.orchestra.run.smithi055.stderr:Cannot find device "ceph-brx" 2024-04-15T15:21:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:21:43.806 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-15T15:21:43.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.911+0000 7f2a7a8f2640 1 Processor -- start 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.911+0000 7f2a7a8f2640 1 -- start start 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a7a8f2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2a7406fd90 0x7f2a740ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a7a8f2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a74071e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a7a8f2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a740aee30 con 0x7f2a74158de0 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a7a8f2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2a740af110 con 0x7f2a7406fd90 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a7a8f2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2a740af3f0 con 0x7f2a740ae600 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a737fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2a7406fd90 0x7f2a740ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a73fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2a74158de0 0x7f2a7406f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44402/0 (socket says 172.21.15.12:44402) 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a73fff640 1 -- 172.21.15.12:0/1604408588 learned_addr learned my addr 172.21.15.12:0/1604408588 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a78e8f640 1 --2- 172.21.15.12:0/1604408588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a74071e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a737fe640 1 -- 172.21.15.12:0/1604408588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 msgr2=0x7f2a74071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a737fe640 1 --2- 172.21.15.12:0/1604408588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a74071e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a737fe640 1 -- 172.21.15.12:0/1604408588 >> v1:172.21.15.12:6789/0 conn(0x7f2a74158de0 legacy=0x7f2a7406f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a737fe640 1 -- 172.21.15.12:0/1604408588 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a740af6d0 con 0x7f2a7406fd90 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a72ffd640 1 -- 172.21.15.12:0/1604408588 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3211480215 0 0) 0x7f2a740aee30 con 0x7f2a74158de0 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.912+0000 7f2a78e8f640 1 --2- 172.21.15.12:0/1604408588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a74071e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.913+0000 7f2a737fe640 1 --2- 172.21.15.12:0/1604408588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a7406fd90 0x7f2a740ae0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800ba40 tx=0x7f2a6802e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d766a472c9d5cfae server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.913+0000 7f2a72ffd640 1 -- 172.21.15.12:0/1604408588 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6803b070 con 0x7f2a7406fd90 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.913+0000 7f2a72ffd640 1 -- 172.21.15.12:0/1604408588 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6802ee30 con 0x7f2a7406fd90 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.913+0000 7f2a72ffd640 1 -- 172.21.15.12:0/1604408588 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68036640 con 0x7f2a7406fd90 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.913+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/1604408588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a7406fd90 msgr2=0x7f2a740ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.913+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/1604408588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a7406fd90 0x7f2a740ae0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800ba40 tx=0x7f2a6802e080 comp rx=0 tx=0).stop 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.914+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/1604408588 shutdown_connections 2024-04-15T15:21:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.914+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/1604408588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a74071e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.914+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/1604408588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a7406fd90 0x7f2a740ae0c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.914+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/1604408588 >> 172.21.15.12:0/1604408588 conn(0x7f2a740b1f60 msgr2=0x7f2a740b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.914+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/1604408588 shutdown_connections 2024-04-15T15:21:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.914+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/1604408588 wait complete. 2024-04-15T15:21:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.915+0000 7f2a7a8f2640 1 Processor -- start 2024-04-15T15:21:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.915+0000 7f2a7a8f2640 1 -- start start 2024-04-15T15:21:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a7a8f2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a7406fd90 0x7f2a740b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a7a8f2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a741523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a7a8f2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a74158de0 0x7f2a741580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a7a8f2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2a741591c0 con 0x7f2a74158de0 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a7a8f2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2a740aeab0 con 0x7f2a7406fd90 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a7a8f2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2a74159340 con 0x7f2a740ae600 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a74158de0 0x7f2a741580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a74158de0 0x7f2a741580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47832/0 (socket says 172.21.15.12:47832) 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 -- 172.21.15.12:0/3343648537 learned_addr learned my addr 172.21.15.12:0/3343648537 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 -- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 msgr2=0x7f2a741523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a737fe640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a741523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a73fff640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a7406fd90 0x7f2a740b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a741523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 -- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a7406fd90 msgr2=0x7f2a740b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a7406fd90 0x7f2a740b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 -- 172.21.15.12:0/3343648537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a74151330 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a78e8f640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a74158de0 0x7f2a741580b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c007f20 tx=0x7f2a6c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a70ff9640 1 -- 172.21.15.12:0/3343648537 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c01a070 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a70ff9640 1 -- 172.21.15.12:0/3343648537 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6c003b90 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a741515f0 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a70ff9640 1 -- 172.21.15.12:0/3343648537 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c004030 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.916+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a74151bb0 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.917+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a740689e0 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.918+0000 7f2a70ff9640 1 -- 172.21.15.12:0/3343648537 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a6c01d070 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.918+0000 7f2a70ff9640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2a6006c8f0 0x7f2a6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.918+0000 7f2a70ff9640 1 -- 172.21.15.12:0/3343648537 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2a6c08c1c0 con 0x7f2a74158de0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.919+0000 7f2a73fff640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2a6006c8f0 0x7f2a6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.919+0000 7f2a73fff640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2a6006c8f0 0x7f2a6006edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2a64007fd0 tx=0x7f2a64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:43.923+0000 7f2a70ff9640 1 -- 172.21.15.12:0/3343648537 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a6c010070 con 0x7f2a74158de0 2024-04-15T15:21:44.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.147+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2a74099e20 con 0x7f2a74158de0 2024-04-15T15:21:44.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.148+0000 7f2a70ff9640 1 -- 172.21.15.12:0/3343648537 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f2a6c05a2a0 con 0x7f2a74158de0 2024-04-15T15:21:44.149 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:21:44.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2a6006c8f0 msgr2=0x7f2a6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:44.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2a6006c8f0 0x7f2a6006edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2a64007fd0 tx=0x7f2a64009040 comp rx=0 tx=0).stop 2024-04-15T15:21:44.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a74158de0 msgr2=0x7f2a741580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:44.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a74158de0 0x7f2a741580b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c007f20 tx=0x7f2a6c00d560 comp rx=0 tx=0).stop 2024-04-15T15:21:44.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 shutdown_connections 2024-04-15T15:21:44.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a74158de0 0x7f2a741580b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:44.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2a740ae600 0x7f2a741523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:44.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2a6006c8f0 0x7f2a6006edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:44.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 --2- 172.21.15.12:0/3343648537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a7406fd90 0x7f2a740b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:44.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 >> 172.21.15.12:0/3343648537 conn(0x7f2a740b1f60 msgr2=0x7f2a7415a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:44.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 shutdown_connections 2024-04-15T15:21:44.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44.150+0000 7f2a7a8f2640 1 -- 172.21.15.12:0/3343648537 wait complete. 2024-04-15T15:21:44.162 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:21:44.162 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:21:44.162 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-04-15T15:21:44.162 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:21:44.162 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:44.162 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:21:44.162 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:21:44.162 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:21:44.163 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:21:44.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:44.163 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-04-15T15:21:44.179 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42644sec preferred_lft 42644sec 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:44.180 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-04-15T15:21:44.180 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:21:44.180 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:44.180 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:44.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:21:44.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-04-15T15:21:44.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-04-15T15:21:44.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:21:44.181 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:44.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:44.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:44.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:44.332 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:21:44.360 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-04-15T15:21:44.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:44.361 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-04-15T15:21:44.417 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-04-15T15:21:44.417 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:21:44.417 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:21:44.417 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:21:44.417 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:21:44.418 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:44.418 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:44.418 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:21:44.418 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:21:44.418 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:21:44.418 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:44.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:44.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:44.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:44.556 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-15T15:21:44.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:44.572 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-04-15T15:21:44.627 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:44.628 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:44.628 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:44.628 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-15T15:21:44.628 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:44.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:44.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:44.753 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-15T15:21:44.753 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:44.753 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:44.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-15T15:21:44.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:21:44.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-15T15:21:44.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-15T15:21:44.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-15T15:21:44.753 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:44.885 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:44.885 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:44.885 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-04-15T15:21:44.885 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:21:44.885 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:44.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:44.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:44.969 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:21:44.969 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:44.969 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:44.984 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:21:44.985 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:44.985 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:45.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:21:45.109 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:45.165 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:21:45.165 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:45.166 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:21:45.167 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:21:45.168 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:21:45.168 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:21:45.168 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:21:45.168 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:45.168 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:45.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:45.168 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:21:45.221 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:21:45.221 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-15T15:21:45.264 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:21:45.291 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:45.349 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:21:45.349 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:21:45.349 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:21:45.350 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:21:45.351 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:21:45.352 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:45.352 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:45.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:45.352 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:21:45.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:2024-04-15T15:21:45.350+0000 7fc06a2da480 -1 init, newargv = 0x5606dd5fe170 newargc=15 2024-04-15T15:21:45.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[47626]: starting ceph client 2024-04-15T15:21:45.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[47626]: starting fuse 2024-04-15T15:21:45.368 INFO:teuthology.orchestra.run.smithi012.stdout:43 2024-04-15T15:21:45.368 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:21:45.368 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:21:45.368 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-04-15T15:21:45.368 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-04-15T15:21:45.368 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-04-15T15:21:45.368 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-04-15T15:21:45.368 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:45.368 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-04-15T15:21:45.368 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-04-15T15:21:45.369 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-04-15T15:21:45.370 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:21:45.370 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:45.370 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.0")) 2024-04-15T15:21:45.370 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-04-15T15:21:45.490 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.0.47626.asok 2024-04-15T15:21:45.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:45.501 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47626.asok 2024-04-15T15:21:45.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:45.501 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47626.asok status 2024-04-15T15:21:45.641 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:21:45.641 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-04-15T15:21:45.641 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "0", 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "47626", 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4842, 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-04-15T15:21:45.642 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4842 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1005118146 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1005118146 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.4842 192.168.0.1:0/1005118146", 2024-04-15T15:21:45.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.1:0/1005118146", 2024-04-15T15:21:45.644 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-04-15T15:21:45.644 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 9, 2024-04-15T15:21:45.644 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 25, 2024-04-15T15:21:45.644 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:21:45.644 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-04-15T15:21:45.644 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-04-15T15:21:45.644 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:21:45.651 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:21:45.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.755+0000 7f9a29455640 1 Processor -- start 2024-04-15T15:21:45.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.755+0000 7f9a29455640 1 -- start start 2024-04-15T15:21:45.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a29455640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9a240d0150 0x7f9a240b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a29455640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9a240b42f0 0x7f9a240d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a29455640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a240d3570 con 0x7f9a24161e50 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a29455640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9a240cf390 con 0x7f9a240d0150 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a29455640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9a240cf670 con 0x7f9a240b42f0 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a1bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9a240d0150 0x7f9a240b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a1bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9a240d0150 0x7f9a240b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47836/0 (socket says 172.21.15.12:47836) 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a1bfff640 1 -- 172.21.15.12:0/2706056629 learned_addr learned my addr 172.21.15.12:0/2706056629 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a237fe640 1 --2- 172.21.15.12:0/2706056629 >> v2:172.21.15.55:3300/0 conn(0x7f9a240b42f0 0x7f9a240d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a1bfff640 1 -- 172.21.15.12:0/2706056629 >> v2:172.21.15.55:3300/0 conn(0x7f9a240b42f0 msgr2=0x7f9a240d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a1bfff640 1 --2- 172.21.15.12:0/2706056629 >> v2:172.21.15.55:3300/0 conn(0x7f9a240b42f0 0x7f9a240d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a1bfff640 1 -- 172.21.15.12:0/2706056629 >> v1:172.21.15.55:6789/0 conn(0x7f9a24161e50 legacy=0x7f9a240d2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:45.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.756+0000 7f9a1bfff640 1 -- 172.21.15.12:0/2706056629 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a240cf950 con 0x7f9a240d0150 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.757+0000 7f9a1bfff640 1 --2- 172.21.15.12:0/2706056629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a240d0150 0x7f9a240b3db0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9a2408e000 tx=0x7f9a0c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e80cb2c3933e7e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.757+0000 7f9a227fc640 1 -- 172.21.15.12:0/2706056629 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c034070 con 0x7f9a240d0150 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.757+0000 7f9a227fc640 1 -- 172.21.15.12:0/2706056629 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a0c030860 con 0x7f9a240d0150 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.757+0000 7f9a227fc640 1 -- 172.21.15.12:0/2706056629 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c030c30 con 0x7f9a240d0150 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.758+0000 7f9a29455640 1 -- 172.21.15.12:0/2706056629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a240d0150 msgr2=0x7f9a240b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.758+0000 7f9a29455640 1 --2- 172.21.15.12:0/2706056629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a240d0150 0x7f9a240b3db0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9a2408e000 tx=0x7f9a0c02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.758+0000 7f9a29455640 1 -- 172.21.15.12:0/2706056629 shutdown_connections 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.758+0000 7f9a29455640 1 --2- 172.21.15.12:0/2706056629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a240d0150 0x7f9a240b3db0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.758+0000 7f9a29455640 1 --2- 172.21.15.12:0/2706056629 >> v2:172.21.15.55:3300/0 conn(0x7f9a240b42f0 0x7f9a240d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.758+0000 7f9a29455640 1 -- 172.21.15.12:0/2706056629 >> 172.21.15.12:0/2706056629 conn(0x7f9a2408bf60 msgr2=0x7f9a240d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.758+0000 7f9a29455640 1 -- 172.21.15.12:0/2706056629 shutdown_connections 2024-04-15T15:21:45.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.758+0000 7f9a29455640 1 -- 172.21.15.12:0/2706056629 wait complete. 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.759+0000 7f9a29455640 1 Processor -- start 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.759+0000 7f9a29455640 1 -- start start 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a29455640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9a240b42f0 0x7f9a240cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a29455640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9a240d0150 0x7f9a240cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a29455640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a24161e50 0x7f9a240cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a29455640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9a240b4900 con 0x7f9a24161e50 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a29455640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9a240b4780 con 0x7f9a240b42f0 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a29455640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9a24162230 con 0x7f9a240d0150 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a24161e50 0x7f9a240cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a24161e50 0x7f9a240cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47848/0 (socket says 172.21.15.12:47848) 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a237fe640 1 -- 172.21.15.12:0/2075084748 learned_addr learned my addr 172.21.15.12:0/2075084748 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:45.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a22ffd640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9a240b42f0 0x7f9a240cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a1bfff640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9a240d0150 0x7f9a240cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a237fe640 1 -- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9a240d0150 msgr2=0x7f9a240cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a237fe640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9a240d0150 0x7f9a240cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a237fe640 1 -- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9a240b42f0 msgr2=0x7f9a240cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a237fe640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9a240b42f0 0x7f9a240cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a237fe640 1 -- 172.21.15.12:0/2075084748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a24160130 con 0x7f9a24161e50 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.760+0000 7f9a22ffd640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9a240b42f0 0x7f9a240cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.761+0000 7f9a237fe640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a24161e50 0x7f9a240cc390 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009dd0 tx=0x7f9a1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.761+0000 7f9a1b7fe640 1 -- 172.21.15.12:0/2075084748 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c018020 con 0x7f9a24161e50 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.761+0000 7f9a1b7fe640 1 -- 172.21.15.12:0/2075084748 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a1c014480 con 0x7f9a24161e50 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.761+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a24160410 con 0x7f9a24161e50 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.761+0000 7f9a1b7fe640 1 -- 172.21.15.12:0/2075084748 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c014870 con 0x7f9a24161e50 2024-04-15T15:21:45.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.761+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a241609d0 con 0x7f9a24161e50 2024-04-15T15:21:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.762+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a240689e0 con 0x7f9a24161e50 2024-04-15T15:21:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.762+0000 7f9a1b7fe640 1 -- 172.21.15.12:0/2075084748 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a1c005d10 con 0x7f9a24161e50 2024-04-15T15:21:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.762+0000 7f9a1b7fe640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9a0406c8f0 0x7f9a0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.763+0000 7f9a1b7fe640 1 -- 172.21.15.12:0/2075084748 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9a1c08ca00 con 0x7f9a24161e50 2024-04-15T15:21:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.763+0000 7f9a22ffd640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9a0406c8f0 0x7f9a0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.763+0000 7f9a22ffd640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9a0406c8f0 0x7f9a0406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9a14001660 tx=0x7f9a14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:45.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.767+0000 7f9a1b7fe640 1 -- 172.21.15.12:0/2075084748 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a1c022070 con 0x7f9a24161e50 2024-04-15T15:21:45.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.992+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9a2415e100 con 0x7f9a24161e50 2024-04-15T15:21:45.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.993+0000 7f9a1b7fe640 1 -- 172.21.15.12:0/2075084748 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f9a1c010040 con 0x7f9a24161e50 2024-04-15T15:21:45.994 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9a0406c8f0 msgr2=0x7f9a0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9a0406c8f0 0x7f9a0406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9a14001660 tx=0x7f9a14009040 comp rx=0 tx=0).stop 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a24161e50 msgr2=0x7f9a240cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a24161e50 0x7f9a240cc390 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009dd0 tx=0x7f9a1c00c490 comp rx=0 tx=0).stop 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 shutdown_connections 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a24161e50 0x7f9a240cc390 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9a240d0150 0x7f9a240cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9a0406c8f0 0x7f9a0406edb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 --2- 172.21.15.12:0/2075084748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9a240b42f0 0x7f9a240cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 >> 172.21.15.12:0/2075084748 conn(0x7f9a2408bf60 msgr2=0x7f9a240d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 shutdown_connections 2024-04-15T15:21:45.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:45.995+0000 7f9a29455640 1 -- 172.21.15.12:0/2075084748 wait complete. 2024-04-15T15:21:46.007 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:21:46.007 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:21:46.008 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-04-15T15:21:46.008 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-15T15:21:46.008 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:21:46.008 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:21:46.008 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:21:46.008 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:21:46.008 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:21:46.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:46.008 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-04-15T15:21:46.024 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:46.024 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:46.024 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:46.024 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.024 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:46.024 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42643sec preferred_lft 42643sec 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:46.025 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 8e:28:9a:60:ef:2c brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::18d2:66ff:fef6:52a/64 scope link 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 8e:28:9a:60:ef:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::8c28:9aff:fe60:ef2c/64 scope link tentative 2024-04-15T15:21:46.026 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:46.027 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:21:46.091 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-04-15T15:21:46.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:46.093 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-04-15T15:21:46.149 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-04-15T15:21:46.149 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:21:46.149 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:21:46.150 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:21:46.150 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:21:46.150 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:46.150 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:46.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:21:46.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:21:46.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:21:46.150 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:46.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:46.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:46.289 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-15T15:21:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:21:46.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:46.305 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-04-15T15:21:46.359 INFO:teuthology.orchestra.run.smithi012.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-15T15:21:46.359 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:46.359 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:46.359 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:21:46.359 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-15T15:21:46.360 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:46.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:46.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:46.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:46.483 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-15T15:21:46.511 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:46.511 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:46.511 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:46.511 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.511 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:46.511 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.512 INFO:teuthology.orchestra.run.smithi012.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:21:46.512 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ae:0e:16:e1:b6:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:21:46.512 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:21:46.512 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.512 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ac0e:16ff:fee1:b6f5/64 scope link tentative 2024-04-15T15:21:46.512 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:46.513 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-15T15:21:46.581 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ae:0e:16:e1:b6:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ac0e:16ff:fee1:b6f5/64 scope link tentative 2024-04-15T15:21:46.582 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:46.583 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-15T15:21:46.584 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:46.584 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:46.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-15T15:21:46.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-15T15:21:46.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-15T15:21:46.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-15T15:21:46.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-15T15:21:46.584 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:46.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:46.754 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:46.754 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:21:46.754 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.1 up 2024-04-15T15:21:46.754 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.1 master ceph-brx 2024-04-15T15:21:46.754 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:21:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:46.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:46.840 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-15T15:21:46.840 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:46.840 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-15T15:21:46.868 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:21:46.868 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:46.868 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:21:46.923 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:21:46.991 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:47.047 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:21:47.048 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:21:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:21:47.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:47.051 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:21:47.103 INFO:teuthology.orchestra.run.smithi012.stdout:43 2024-04-15T15:21:47.103 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:21:47.103 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-15T15:21:47.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:21:47.175 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:47.231 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:21:47.231 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:21:47.231 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:21:47.231 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:21:47.231 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:21:47.231 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:21:47.231 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:47.232 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:21:47.233 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:21:47.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:47.234 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:21:47.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:2024-04-15T15:21:47.234+0000 7f89d6e3d480 -1 init, newargv = 0x558748f3aea0 newargc=15 2024-04-15T15:21:47.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:ceph-fuse[48124]: starting ceph client 2024-04-15T15:21:47.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:ceph-fuse[48124]: starting fuse 2024-04-15T15:21:47.250 INFO:teuthology.orchestra.run.smithi012.stdout:43 2024-04-15T15:21:47.251 INFO:teuthology.orchestra.run.smithi012.stdout:44 2024-04-15T15:21:47.251 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:21:47.251 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:21:47.251 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-04-15T15:21:47.251 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-04-15T15:21:47.251 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:21:47.252 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:21:47.253 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:21:47.253 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-04-15T15:21:47.253 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-04-15T15:21:47.253 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-04-15T15:21:47.253 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:21:47.253 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:21:47.253 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.2")) 2024-04-15T15:21:47.253 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-04-15T15:21:47.374 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.2.48124.asok 2024-04-15T15:21:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:47.384 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.48124.asok 2024-04-15T15:21:47.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:47.384 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.48124.asok status 2024-04-15T15:21:47.524 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:21:47.524 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-04-15T15:21:47.524 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:21:47.524 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:21:47.524 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "2", 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "48124", 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 4854, 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4854 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:47.525 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2125463221 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2125463221 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.4854 192.168.0.2:0/2125463221", 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.2:0/2125463221", 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-04-15T15:21:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 9, 2024-04-15T15:21:47.527 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 25, 2024-04-15T15:21:47.527 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:21:47.527 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-04-15T15:21:47.527 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-04-15T15:21:47.527 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:21:47.532 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:21:47.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.635+0000 7f577c4b1640 1 Processor -- start 2024-04-15T15:21:47.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.635+0000 7f577c4b1640 1 -- start start 2024-04-15T15:21:47.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.636+0000 7f577c4b1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f577415d270 0x7f577415d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:47.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.636+0000 7f577c4b1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57740cd160 con 0x7f57740b3a30 2024-04-15T15:21:47.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.636+0000 7f577c4b1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57740cd440 con 0x7f577406b9e0 2024-04-15T15:21:47.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.636+0000 7f577c4b1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f57740cd720 con 0x7f577415d270 2024-04-15T15:21:47.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.636+0000 7f577aa4e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f577415d270 0x7f577415d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:47.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.636+0000 7f5779a4c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f57740b3a30 0x7f577415ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44406/0 (socket says 172.21.15.12:44406) 2024-04-15T15:21:47.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.636+0000 7f5779a4c640 1 -- 172.21.15.12:0/1704973314 learned_addr learned my addr 172.21.15.12:0/1704973314 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:47.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.637+0000 7f577aa4e640 1 -- 172.21.15.12:0/1704973314 >> v1:172.21.15.55:6789/0 conn(0x7f577406b9e0 legacy=0x7f577406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:47.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.637+0000 7f577aa4e640 1 -- 172.21.15.12:0/1704973314 >> v1:172.21.15.12:6789/0 conn(0x7f57740b3a30 legacy=0x7f577415ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:47.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.637+0000 7f577aa4e640 1 -- 172.21.15.12:0/1704973314 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57740cda00 con 0x7f577415d270 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.637+0000 7f577aa4e640 1 --2- 172.21.15.12:0/1704973314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f577415d6c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5764009d90 tx=0x7f576402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46029f1f14302a5f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577924b640 1 -- 172.21.15.12:0/1704973314 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764039070 con 0x7f577415d270 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577924b640 1 -- 172.21.15.12:0/1704973314 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f576402cd00 con 0x7f577415d270 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577924b640 1 -- 172.21.15.12:0/1704973314 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764034660 con 0x7f577415d270 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577c4b1640 1 -- 172.21.15.12:0/1704973314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 msgr2=0x7f577415d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577c4b1640 1 --2- 172.21.15.12:0/1704973314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f577415d6c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5764009d90 tx=0x7f576402bf50 comp rx=0 tx=0).stop 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577c4b1640 1 -- 172.21.15.12:0/1704973314 shutdown_connections 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577c4b1640 1 --2- 172.21.15.12:0/1704973314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f577415d6c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577c4b1640 1 -- 172.21.15.12:0/1704973314 >> 172.21.15.12:0/1704973314 conn(0x7f577409a9d0 msgr2=0x7f57740b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:47.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577c4b1640 1 -- 172.21.15.12:0/1704973314 shutdown_connections 2024-04-15T15:21:47.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.638+0000 7f577c4b1640 1 -- 172.21.15.12:0/1704973314 wait complete. 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.639+0000 7f577c4b1640 1 Processor -- start 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.639+0000 7f577c4b1640 1 -- start start 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577c4b1640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f577406b9e0 0x7f57740b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577c4b1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57740b3a30 0x7f57740c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577c4b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f5774156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577c4b1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f57740ce200 con 0x7f577415d270 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577c4b1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f57740cce50 con 0x7f57740b3a30 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577c4b1640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f57740ce380 con 0x7f577406b9e0 2024-04-15T15:21:47.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577aa4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f5774156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577aa4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f5774156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47866/0 (socket says 172.21.15.12:47866) 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577aa4e640 1 -- 172.21.15.12:0/1509359605 learned_addr learned my addr 172.21.15.12:0/1509359605 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f5779a4c640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57740b3a30 0x7f57740c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.640+0000 7f577a24d640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f577406b9e0 0x7f57740b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577aa4e640 1 -- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f577406b9e0 msgr2=0x7f57740b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577aa4e640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f577406b9e0 0x7f57740b6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577aa4e640 1 -- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57740b3a30 msgr2=0x7f57740c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577aa4e640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57740b3a30 0x7f57740c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577aa4e640 1 -- 172.21.15.12:0/1509359605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57740c8d40 con 0x7f577415d270 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577a24d640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f577406b9e0 0x7f57740b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577aa4e640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f5774156830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5764009eb0 tx=0x7f5764004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f5772ffd640 1 -- 172.21.15.12:0/1509359605 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764039050 con 0x7f577415d270 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f5772ffd640 1 -- 172.21.15.12:0/1509359605 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5764045070 con 0x7f577415d270 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57740c9020 con 0x7f577415d270 2024-04-15T15:21:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f5772ffd640 1 -- 172.21.15.12:0/1509359605 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576403e850 con 0x7f577415d270 2024-04-15T15:21:47.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.641+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57740c95e0 con 0x7f577415d270 2024-04-15T15:21:47.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.642+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57740690f0 con 0x7f577415d270 2024-04-15T15:21:47.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.642+0000 7f5772ffd640 1 -- 172.21.15.12:0/1509359605 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f576403e9f0 con 0x7f577415d270 2024-04-15T15:21:47.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.643+0000 7f5772ffd640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f576006c8f0 0x7f576006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:47.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.643+0000 7f5772ffd640 1 -- 172.21.15.12:0/1509359605 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f57640aeb50 con 0x7f577415d270 2024-04-15T15:21:47.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.643+0000 7f577a24d640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f576006c8f0 0x7f576006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:47.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.643+0000 7f577a24d640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f576006c8f0 0x7f576006edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f576c001700 tx=0x7f576c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:47.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.647+0000 7f5772ffd640 1 -- 172.21.15.12:0/1509359605 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f576407cd00 con 0x7f577415d270 2024-04-15T15:21:47.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.872+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5774073200 con 0x7f577415d270 2024-04-15T15:21:47.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.872+0000 7f5772ffd640 1 -- 172.21.15.12:0/1509359605 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f576407c600 con 0x7f577415d270 2024-04-15T15:21:47.874 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:21:47.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f576006c8f0 msgr2=0x7f576006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f576006c8f0 0x7f576006edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f576c001700 tx=0x7f576c009040 comp rx=0 tx=0).stop 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 msgr2=0x7f5774156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f5774156830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5764009eb0 tx=0x7f5764004a40 comp rx=0 tx=0).stop 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 shutdown_connections 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f577415d270 0x7f5774156830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f576006c8f0 0x7f576006edb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57740b3a30 0x7f57740c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 --2- 172.21.15.12:0/1509359605 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f577406b9e0 0x7f57740b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 >> 172.21.15.12:0/1509359605 conn(0x7f577409a9d0 msgr2=0x7f5774160910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 shutdown_connections 2024-04-15T15:21:47.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:47.874+0000 7f577c4b1640 1 -- 172.21.15.12:0/1509359605 wait complete. 2024-04-15T15:21:47.886 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:21:47.886 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:21:47.886 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi055.front.sepia.ceph.com 2024-04-15T15:21:47.886 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:21:47.886 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:47.886 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:21:47.887 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:21:47.887 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:21:47.887 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:21:47.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:47.887 DEBUG:teuthology.orchestra.run.smithi055:> ip addr 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:47.907 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f0 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f1 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:06 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f0 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.21.15.55/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft 42681sec preferred_lft 42681sec 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:07 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:47.908 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f1 2024-04-15T15:21:47.909 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:21:47.909 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:47.909 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:47.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:21:47.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip addr flush dev ceph-brx 2024-04-15T15:21:47.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set ceph-brx up 2024-04-15T15:21:47.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:21:47.909 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:48.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:48.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:48.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:48.122 DEBUG:teuthology.orchestra.run.smithi055:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:21:48.151 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-04-15T15:21:48.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:48.152 DEBUG:teuthology.orchestra.run.smithi055:> route 2024-04-15T15:21:48.223 INFO:teuthology.orchestra.run.smithi055.stdout:Kernel IP routing table 2024-04-15T15:21:48.223 INFO:teuthology.orchestra.run.smithi055.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:21:48.223 INFO:teuthology.orchestra.run.smithi055.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:21:48.223 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:21:48.223 INFO:teuthology.orchestra.run.smithi055.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:21:48.223 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:48.224 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:48.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:21:48.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:21:48.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:21:48.224 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:48.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:48.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:48.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:48.544 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-04-15T15:21:48.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:48.559 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list-id 2024-04-15T15:21:48.615 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:48.615 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:48.615 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:48.615 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-15T15:21:48.615 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:48.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:48.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:48.744 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-15T15:21:48.744 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:48.744 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:48.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-15T15:21:48.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:21:48.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-15T15:21:48.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-15T15:21:48.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-15T15:21:48.744 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:48.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:48.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:48.889 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:48.889 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:48.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set brx.0 up 2024-04-15T15:21:48.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:21:48.890 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:48.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:48.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:48.974 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:21:48.975 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:48.975 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:48.989 INFO:teuthology.orchestra.run.smithi055.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:21:48.990 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:48.990 DEBUG:teuthology.orchestra.run.smithi055:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:49.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:21:49.115 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:49.171 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:21:49.171 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:21:49.171 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:21:49.171 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:21:49.172 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:21:49.173 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:49.174 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:49.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:49.174 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:21:49.226 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:21:49.226 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-15T15:21:49.268 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:21:49.298 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:49.354 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:21:49.354 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:21:49.354 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:21:49.354 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:21:49.354 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:21:49.354 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:21:49.354 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:21:49.354 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:49.355 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:49.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:49.357 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:21:49.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:2024-04-15T15:21:49.358+0000 7f4b1720d480 -1 init, newargv = 0x5557becc2020 newargc=15 2024-04-15T15:21:49.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:ceph-fuse[44424]: starting ceph client 2024-04-15T15:21:49.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:ceph-fuse[44424]: starting fuse 2024-04-15T15:21:49.376 INFO:teuthology.orchestra.run.smithi055.stdout:41 2024-04-15T15:21:49.376 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-15T15:21:49.376 DEBUG:teuthology.orchestra.run.smithi055:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:21:49.376 DEBUG:teuthology.orchestra.run.smithi055:> import glob 2024-04-15T15:21:49.376 DEBUG:teuthology.orchestra.run.smithi055:> import re 2024-04-15T15:21:49.376 DEBUG:teuthology.orchestra.run.smithi055:> import os 2024-04-15T15:21:49.376 DEBUG:teuthology.orchestra.run.smithi055:> import subprocess 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> def _find_admin_socket(client_name): 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> files = glob.glob(asok_path) 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> # Given a non-glob path, it better be there 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> if "*" not in asok_path: 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> assert(len(files) == 1) 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> return files[0] 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> for f in files: 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:21:49.377 DEBUG:teuthology.orchestra.run.smithi055:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:21:49.378 DEBUG:teuthology.orchestra.run.smithi055:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:21:49.378 DEBUG:teuthology.orchestra.run.smithi055:> contents = proc_f.read() 2024-04-15T15:21:49.378 DEBUG:teuthology.orchestra.run.smithi055:> if mountpoint in contents: 2024-04-15T15:21:49.378 DEBUG:teuthology.orchestra.run.smithi055:> return f 2024-04-15T15:21:49.378 DEBUG:teuthology.orchestra.run.smithi055:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:21:49.378 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:21:49.378 DEBUG:teuthology.orchestra.run.smithi055:> print(_find_admin_socket("client.1")) 2024-04-15T15:21:49.378 DEBUG:teuthology.orchestra.run.smithi055:> ' 2024-04-15T15:21:49.501 INFO:teuthology.orchestra.run.smithi055.stdout:/var/run/ceph/ceph-client.1.44424.asok 2024-04-15T15:21:49.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:49.512 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44424.asok 2024-04-15T15:21:49.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:49.512 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44424.asok status 2024-04-15T15:21:49.653 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-15T15:21:49.653 INFO:teuthology.orchestra.run.smithi055.stdout: "metadata": { 2024-04-15T15:21:49.653 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:21:49.653 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:21:49.653 INFO:teuthology.orchestra.run.smithi055.stdout: "entity_id": "1", 2024-04-15T15:21:49.653 INFO:teuthology.orchestra.run.smithi055.stdout: "hostname": "smithi055", 2024-04-15T15:21:49.653 INFO:teuthology.orchestra.run.smithi055.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:21:49.653 INFO:teuthology.orchestra.run.smithi055.stdout: "pid": "44424", 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "root": "/" 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_count": 0, 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_pinned_count": 0, 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "id": 4732, 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "inst": { 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "name": { 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "client", 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "num": 4732 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 4072169530 2024-04-15T15:21:49.654 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 4072169530 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "inst_str": "client.4732 192.168.0.1:0/4072169530", 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "addr_str": "192.168.0.1:0/4072169530", 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "inode_count": 1, 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_epoch": 9, 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch": 25, 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklisted": false, 2024-04-15T15:21:49.655 INFO:teuthology.orchestra.run.smithi055.stdout: "fs_name": "cephfs" 2024-04-15T15:21:49.656 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-15T15:21:49.661 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:21:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.776+0000 7f39d1e0d640 1 Processor -- start 2024-04-15T15:21:49.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.776+0000 7f39d1e0d640 1 -- start start 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39d1e0d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc161e50 0x7f39cc0d2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39d1e0d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f39cc0d0150 0x7f39cc0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39d1e0d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39cc0d3570 con 0x7f39cc0d05b0 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39d1e0d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f39cc0cf390 con 0x7f39cc0d0150 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39d1e0d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f39cc0cf670 con 0x7f39cc161e50 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39caffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f39cc0d0150 0x7f39cc0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39cb7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc161e50 0x7f39cc0d2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39cbfff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f39cc0d05b0 0x7f39cc0b42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:58682/0 (socket says 172.21.15.12:58682) 2024-04-15T15:21:49.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.777+0000 7f39cbfff640 1 -- 172.21.15.12:0/114152110 learned_addr learned my addr 172.21.15.12:0/114152110 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.778+0000 7f39cb7fe640 1 -- 172.21.15.12:0/114152110 >> v2:172.21.15.55:3300/0 conn(0x7f39cc0d0150 msgr2=0x7f39cc0b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.778+0000 7f39cb7fe640 1 --2- 172.21.15.12:0/114152110 >> v2:172.21.15.55:3300/0 conn(0x7f39cc0d0150 0x7f39cc0b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.778+0000 7f39cb7fe640 1 -- 172.21.15.12:0/114152110 >> v1:172.21.15.55:6789/0 conn(0x7f39cc0d05b0 legacy=0x7f39cc0b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.778+0000 7f39cb7fe640 1 -- 172.21.15.12:0/114152110 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc0cf950 con 0x7f39cc161e50 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.778+0000 7f39ca7fc640 1 -- 172.21.15.12:0/114152110 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827824421 0 0) 0x7f39cc0d3570 con 0x7f39cc0d05b0 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.778+0000 7f39caffd640 1 --2- 172.21.15.12:0/114152110 >> v2:172.21.15.55:3300/0 conn(0x7f39cc0d0150 0x7f39cc0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.779+0000 7f39cb7fe640 1 --2- 172.21.15.12:0/114152110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc161e50 0x7f39cc0d2d20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0097a0 tx=0x7f39bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef558a5c748b1bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.779+0000 7f39ca7fc640 1 -- 172.21.15.12:0/114152110 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc035020 con 0x7f39cc161e50 2024-04-15T15:21:49.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.779+0000 7f39ca7fc640 1 -- 172.21.15.12:0/114152110 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39bc031830 con 0x7f39cc161e50 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.779+0000 7f39ca7fc640 1 -- 172.21.15.12:0/114152110 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc031c00 con 0x7f39cc161e50 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.779+0000 7f39d1e0d640 1 -- 172.21.15.12:0/114152110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc161e50 msgr2=0x7f39cc0d2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.779+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/114152110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc161e50 0x7f39cc0d2d20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0097a0 tx=0x7f39bc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.780+0000 7f39d1e0d640 1 -- 172.21.15.12:0/114152110 shutdown_connections 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.780+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/114152110 >> v2:172.21.15.55:3300/0 conn(0x7f39cc0d0150 0x7f39cc0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.780+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/114152110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc161e50 0x7f39cc0d2d20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.780+0000 7f39d1e0d640 1 -- 172.21.15.12:0/114152110 >> 172.21.15.12:0/114152110 conn(0x7f39cc08bf60 msgr2=0x7f39cc0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.780+0000 7f39d1e0d640 1 -- 172.21.15.12:0/114152110 shutdown_connections 2024-04-15T15:21:49.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.780+0000 7f39d1e0d640 1 -- 172.21.15.12:0/114152110 wait complete. 2024-04-15T15:21:49.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.781+0000 7f39d1e0d640 1 Processor -- start 2024-04-15T15:21:49.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.781+0000 7f39d1e0d640 1 -- start start 2024-04-15T15:21:49.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39d1e0d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39cc0d0150 0x7f39cc1f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39d1e0d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39cc161e50 0x7f39cc0b9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39d1e0d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc15e5b0 0x7f39cc1f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39d1e0d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc162230 con 0x7f39cc161e50 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39d1e0d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc0d3260 con 0x7f39cc0d0150 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39d1e0d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f39cc1623b0 con 0x7f39cc15e5b0 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39cc161e50 0x7f39cc0b9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39cc161e50 0x7f39cc0b9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47878/0 (socket says 172.21.15.12:47878) 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 -- 172.21.15.12:0/2839319010 learned_addr learned my addr 172.21.15.12:0/2839319010 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39cbfff640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc15e5b0 0x7f39cc1f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39cb7fe640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39cc0d0150 0x7f39cc1f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 -- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc15e5b0 msgr2=0x7f39cc1f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc15e5b0 0x7f39cc1f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 -- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39cc0d0150 msgr2=0x7f39cc1f48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:49.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39cc0d0150 0x7f39cc1f48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 -- 172.21.15.12:0/2839319010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc0b7310 con 0x7f39cc161e50 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39cb7fe640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39cc0d0150 0x7f39cc1f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39cbfff640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc15e5b0 0x7f39cc1f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.782+0000 7f39caffd640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39cc161e50 0x7f39cc0b9d80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f39c000e960 tx=0x7f39c000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.783+0000 7f39b3fff640 1 -- 172.21.15.12:0/2839319010 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0017070 con 0x7f39cc161e50 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.783+0000 7f39b3fff640 1 -- 172.21.15.12:0/2839319010 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c000ca00 con 0x7f39cc161e50 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.783+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39cc15ed90 con 0x7f39cc161e50 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.783+0000 7f39b3fff640 1 -- 172.21.15.12:0/2839319010 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c000cdd0 con 0x7f39cc161e50 2024-04-15T15:21:49.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.783+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39cc0b7890 con 0x7f39cc161e50 2024-04-15T15:21:49.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.784+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39cc0689e0 con 0x7f39cc161e50 2024-04-15T15:21:49.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.784+0000 7f39b3fff640 1 -- 172.21.15.12:0/2839319010 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f39c001a070 con 0x7f39cc161e50 2024-04-15T15:21:49.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.784+0000 7f39b3fff640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f39b806c920 0x7f39b806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:49.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.785+0000 7f39b3fff640 1 -- 172.21.15.12:0/2839319010 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f39c008bd20 con 0x7f39cc161e50 2024-04-15T15:21:49.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.785+0000 7f39cb7fe640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f39b806c920 0x7f39b806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:49.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.785+0000 7f39cb7fe640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f39b806c920 0x7f39b806ede0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39bc036000 tx=0x7f39bc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:49.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:49.789+0000 7f39b3fff640 1 -- 172.21.15.12:0/2839319010 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39c0059e10 con 0x7f39cc161e50 2024-04-15T15:21:50.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.013+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f39cc15f0d0 con 0x7f39cc161e50 2024-04-15T15:21:50.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.013+0000 7f39b3fff640 1 -- 172.21.15.12:0/2839319010 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f39c0059710 con 0x7f39cc161e50 2024-04-15T15:21:50.015 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f39b806c920 msgr2=0x7f39b806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f39b806c920 0x7f39b806ede0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39bc036000 tx=0x7f39bc008030 comp rx=0 tx=0).stop 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39cc161e50 msgr2=0x7f39cc0b9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39cc161e50 0x7f39cc0b9d80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f39c000e960 tx=0x7f39c000ee20 comp rx=0 tx=0).stop 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 shutdown_connections 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f39cc15e5b0 0x7f39cc1f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39cc161e50 0x7f39cc0b9d80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f39b806c920 0x7f39b806ede0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 --2- 172.21.15.12:0/2839319010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39cc0d0150 0x7f39cc1f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 >> 172.21.15.12:0/2839319010 conn(0x7f39cc08bf60 msgr2=0x7f39cc06ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:50.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 shutdown_connections 2024-04-15T15:21:50.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:50.015+0000 7f39d1e0d640 1 -- 172.21.15.12:0/2839319010 wait complete. 2024-04-15T15:21:50.027 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:21:50.027 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:21:50.028 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi055.front.sepia.ceph.com 2024-04-15T15:21:50.028 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-15T15:21:50.028 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:21:50.028 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:21:50.028 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:21:50.028 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:21:50.028 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:21:50.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:50.028 DEBUG:teuthology.orchestra.run.smithi055:> ip addr 2024-04-15T15:21:50.046 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:50.046 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:50.046 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:50.046 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f0 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f1 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:06 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f0 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.21.15.55/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:21:50.047 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft 42679sec preferred_lft 42679sec 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:07 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f1 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 2a:9f:ac:c6:f4:d0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::9cf3:7dff:fe9f:3592/64 scope link 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 2a:9f:ac:c6:f4:d0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::289f:acff:fec6:f4d0/64 scope link tentative 2024-04-15T15:21:50.048 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:50.049 DEBUG:teuthology.orchestra.run.smithi055:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:21:50.115 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-04-15T15:21:50.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:50.117 DEBUG:teuthology.orchestra.run.smithi055:> route 2024-04-15T15:21:50.175 INFO:teuthology.orchestra.run.smithi055.stdout:Kernel IP routing table 2024-04-15T15:21:50.175 INFO:teuthology.orchestra.run.smithi055.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:21:50.175 INFO:teuthology.orchestra.run.smithi055.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:21:50.175 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:21:50.175 INFO:teuthology.orchestra.run.smithi055.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:21:50.175 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:50.175 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:50.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:21:50.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:21:50.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:21:50.176 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:50.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:50.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:50.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:50.318 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-04-15T15:21:50.333 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:21:50.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:50.334 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list-id 2024-04-15T15:21:50.389 INFO:teuthology.orchestra.run.smithi055.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-15T15:21:50.389 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:50.389 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:50.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:21:50.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-15T15:21:50.389 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:50.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:50.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:50.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:50.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-15T15:21:50.542 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:50.542 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:50.542 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:50.542 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.542 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:50.543 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.543 INFO:teuthology.orchestra.run.smithi055.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:21:50.543 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether a6:84:93:73:da:a0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:21:50.543 INFO:teuthology.orchestra.run.smithi055.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:21:50.543 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.543 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::a484:93ff:fe73:daa0/64 scope link tentative 2024-04-15T15:21:50.543 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:50.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-15T15:21:50.612 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:21:50.613 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:21:50.613 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:21:50.613 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.613 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-04-15T15:21:50.613 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.613 INFO:teuthology.orchestra.run.smithi055.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:21:50.613 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether a6:84:93:73:da:a0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:21:50.614 INFO:teuthology.orchestra.run.smithi055.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:21:50.614 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.614 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::a484:93ff:fe73:daa0/64 scope link tentative 2024-04-15T15:21:50.614 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:21:50.615 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-15T15:21:50.615 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:50.615 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:50.615 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-15T15:21:50.615 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-15T15:21:50.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-15T15:21:50.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-15T15:21:50.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-15T15:21:50.616 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:50.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:50.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:50.788 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:21:50.788 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:21:50.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set brx.1 up 2024-04-15T15:21:50.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set dev brx.1 master ceph-brx 2024-04-15T15:21:50.788 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:21:50.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:21:50.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:50.878 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-15T15:21:50.878 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:50.878 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-15T15:21:50.903 INFO:teuthology.orchestra.run.smithi055.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:21:50.903 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:21:50.903 DEBUG:teuthology.orchestra.run.smithi055:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:21:50.958 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:21:51.027 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:51.084 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:21:51.084 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:21:51.084 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:21:51.084 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:21:51.085 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:51.086 INFO:teuthology.orchestra.run.smithi055.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:51.087 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:21:51.087 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:21:51.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.087 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:21:51.139 INFO:teuthology.orchestra.run.smithi055.stdout:41 2024-04-15T15:21:51.140 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-04-15T15:21:51.140 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-15T15:21:51.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:21:51.210 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:21:51.266 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:21:51.266 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:21:51.266 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:21:51.266 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:21:51.267 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:21:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:51.269 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:21:51.269 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:21:51.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.269 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:21:51.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi055.stderr:2024-04-15T15:21:51.269+0000 7fe7b5666480 -1 init, newargv = 0x5614bb8ad940 newargc=15 2024-04-15T15:21:51.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi055.stderr:ceph-fuse[44881]: starting ceph client 2024-04-15T15:21:51.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi055.stderr:ceph-fuse[44881]: starting fuse 2024-04-15T15:21:51.287 INFO:teuthology.orchestra.run.smithi055.stdout:41 2024-04-15T15:21:51.287 INFO:teuthology.orchestra.run.smithi055.stdout:44 2024-04-15T15:21:51.287 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-04-15T15:21:51.287 DEBUG:teuthology.orchestra.run.smithi055:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:21:51.287 DEBUG:teuthology.orchestra.run.smithi055:> import glob 2024-04-15T15:21:51.287 DEBUG:teuthology.orchestra.run.smithi055:> import re 2024-04-15T15:21:51.287 DEBUG:teuthology.orchestra.run.smithi055:> import os 2024-04-15T15:21:51.287 DEBUG:teuthology.orchestra.run.smithi055:> import subprocess 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> def _find_admin_socket(client_name): 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> files = glob.glob(asok_path) 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> # Given a non-glob path, it better be there 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> if "*" not in asok_path: 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> assert(len(files) == 1) 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> return files[0] 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> for f in files: 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> contents = proc_f.read() 2024-04-15T15:21:51.288 DEBUG:teuthology.orchestra.run.smithi055:> if mountpoint in contents: 2024-04-15T15:21:51.289 DEBUG:teuthology.orchestra.run.smithi055:> return f 2024-04-15T15:21:51.289 DEBUG:teuthology.orchestra.run.smithi055:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:21:51.289 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:21:51.289 DEBUG:teuthology.orchestra.run.smithi055:> print(_find_admin_socket("client.3")) 2024-04-15T15:21:51.289 DEBUG:teuthology.orchestra.run.smithi055:> ' 2024-04-15T15:21:51.410 INFO:teuthology.orchestra.run.smithi055.stdout:/var/run/ceph/ceph-client.3.44881.asok 2024-04-15T15:21:51.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:21:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:21:51.420 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44881.asok 2024-04-15T15:21:51.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44881.asok status 2024-04-15T15:21:51.559 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "metadata": { 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "entity_id": "3", 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "hostname": "smithi055", 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "pid": "44881", 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "root": "/" 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_count": 0, 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_pinned_count": 0, 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "id": 4750, 2024-04-15T15:21:51.560 INFO:teuthology.orchestra.run.smithi055.stdout: "inst": { 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "name": { 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "client", 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "num": 4750 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3037775300 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3037775300 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:21:51.561 INFO:teuthology.orchestra.run.smithi055.stdout: "inst_str": "client.4750 192.168.0.2:0/3037775300", 2024-04-15T15:21:51.562 INFO:teuthology.orchestra.run.smithi055.stdout: "addr_str": "192.168.0.2:0/3037775300", 2024-04-15T15:21:51.562 INFO:teuthology.orchestra.run.smithi055.stdout: "inode_count": 1, 2024-04-15T15:21:51.562 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_epoch": 9, 2024-04-15T15:21:51.562 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch": 26, 2024-04-15T15:21:51.562 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:21:51.562 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklisted": false, 2024-04-15T15:21:51.562 INFO:teuthology.orchestra.run.smithi055.stdout: "fs_name": "cephfs" 2024-04-15T15:21:51.562 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-15T15:21:51.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.568 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-04-15T15:21:51.568 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:51.587 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-04-15T15:21:51.588 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:51.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.588 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:21:51.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.662 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-04-15T15:21:51.662 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-15T15:21:51.720 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-04-15T15:21:51.720 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-15T15:21:51.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.720 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:21:51.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.791 DEBUG:teuthology.orchestra.run.smithi055:> stat --file-system '--printf=%T 2024-04-15T15:21:51.791 DEBUG:teuthology.orchestra.run.smithi055:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:51.810 INFO:teuthology.orchestra.run.smithi055.stdout:fuseblk 2024-04-15T15:21:51.810 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:51.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:21:51.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.883 DEBUG:teuthology.orchestra.run.smithi055:> stat --file-system '--printf=%T 2024-04-15T15:21:51.883 DEBUG:teuthology.orchestra.run.smithi055:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-15T15:21:51.939 INFO:teuthology.orchestra.run.smithi055.stdout:fuseblk 2024-04-15T15:21:51.940 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-15T15:21:51.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:21:51.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:21:52.009 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-15T15:21:52.027 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.136+0000 7ff379731640 1 Processor -- start 2024-04-15T15:21:52.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.136+0000 7ff379731640 1 -- start start 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.137+0000 7ff379731640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff37415c8d0 0x7ff3740cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.137+0000 7ff379731640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409b650 0x7ff37409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.137+0000 7ff379731640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3740d0350 con 0x7ff37409af10 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.137+0000 7ff379731640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff3740b5b00 con 0x7ff37415c8d0 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.137+0000 7ff379731640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff3740ce100 con 0x7ff37409b650 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.137+0000 7ff372ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff37415c8d0 0x7ff3740cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.137+0000 7ff372ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff37415c8d0 0x7ff3740cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47890/0 (socket says 172.21.15.12:47890) 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.137+0000 7ff372ffd640 1 -- 172.21.15.12:0/2178780089 learned_addr learned my addr 172.21.15.12:0/2178780089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.138+0000 7ff3727fc640 1 --2- 172.21.15.12:0/2178780089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409b650 0x7ff37409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.138+0000 7ff372ffd640 1 -- 172.21.15.12:0/2178780089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409b650 msgr2=0x7ff37409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.138+0000 7ff372ffd640 1 --2- 172.21.15.12:0/2178780089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409b650 0x7ff37409a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.138+0000 7ff372ffd640 1 -- 172.21.15.12:0/2178780089 >> v1:172.21.15.55:6789/0 conn(0x7ff37409af10 legacy=0x7ff3740d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.138+0000 7ff372ffd640 1 -- 172.21.15.12:0/2178780089 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3740b54b0 con 0x7ff37415c8d0 2024-04-15T15:21:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.138+0000 7ff371ffb640 1 -- 172.21.15.12:0/2178780089 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 306812555 0 0) 0x7ff3740d0350 con 0x7ff37409af10 2024-04-15T15:21:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff372ffd640 1 --2- 172.21.15.12:0/2178780089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff37415c8d0 0x7ff3740cda70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff364009e50 tx=0x7ff36402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a86f9cd1d41cf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff371ffb640 1 -- 172.21.15.12:0/2178780089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36402ca70 con 0x7ff37415c8d0 2024-04-15T15:21:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff371ffb640 1 -- 172.21.15.12:0/2178780089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36402cc10 con 0x7ff37415c8d0 2024-04-15T15:21:52.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff371ffb640 1 -- 172.21.15.12:0/2178780089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3640343f0 con 0x7ff37415c8d0 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff379731640 1 -- 172.21.15.12:0/2178780089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff37415c8d0 msgr2=0x7ff3740cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff379731640 1 --2- 172.21.15.12:0/2178780089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff37415c8d0 0x7ff3740cda70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff364009e50 tx=0x7ff36402bed0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff379731640 1 -- 172.21.15.12:0/2178780089 shutdown_connections 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff379731640 1 --2- 172.21.15.12:0/2178780089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409b650 0x7ff37409a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff379731640 1 --2- 172.21.15.12:0/2178780089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff37415c8d0 0x7ff3740cda70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff379731640 1 -- 172.21.15.12:0/2178780089 >> 172.21.15.12:0/2178780089 conn(0x7ff3740b3fb0 msgr2=0x7ff3740b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff379731640 1 -- 172.21.15.12:0/2178780089 shutdown_connections 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.139+0000 7ff379731640 1 -- 172.21.15.12:0/2178780089 wait complete. 2024-04-15T15:21:52.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.141+0000 7ff379731640 1 Processor -- start 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.141+0000 7ff379731640 1 -- start start 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.141+0000 7ff379731640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409af10 0x7ff37414f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.141+0000 7ff379731640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff37409b650 0x7ff3740ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.141+0000 7ff379731640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3740cf410 0x7ff37406caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.141+0000 7ff379731640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff3740b7120 con 0x7ff3740cf410 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.141+0000 7ff379731640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff3740b6fa0 con 0x7ff37409b650 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.141+0000 7ff379731640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff3740b72a0 con 0x7ff37409af10 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3740cf410 0x7ff37406caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3740cf410 0x7ff37406caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47896/0 (socket says 172.21.15.12:47896) 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 -- 172.21.15.12:0/1877227569 learned_addr learned my addr 172.21.15.12:0/1877227569 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3727fc640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff37409b650 0x7ff3740ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff372ffd640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409af10 0x7ff37414f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 -- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409af10 msgr2=0x7ff37414f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409af10 0x7ff37414f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 -- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff37409b650 msgr2=0x7ff3740ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff37409b650 0x7ff3740ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 -- 172.21.15.12:0/1877227569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37414ff00 con 0x7ff3740cf410 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff372ffd640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409af10 0x7ff37414f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff3737fe640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3740cf410 0x7ff37406caf0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00c390 tx=0x7ff36c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff36b7fe640 1 -- 172.21.15.12:0/1877227569 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c017070 con 0x7ff3740cf410 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff36b7fe640 1 -- 172.21.15.12:0/1877227569 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36c0035f0 con 0x7ff3740cf410 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37406d7e0 con 0x7ff3740cf410 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff36b7fe640 1 -- 172.21.15.12:0/1877227569 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c0039e0 con 0x7ff3740cf410 2024-04-15T15:21:52.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.142+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37406dda0 con 0x7ff3740cf410 2024-04-15T15:21:52.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.143+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3740690f0 con 0x7ff3740cf410 2024-04-15T15:21:52.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.144+0000 7ff36b7fe640 1 -- 172.21.15.12:0/1877227569 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff36c01a070 con 0x7ff3740cf410 2024-04-15T15:21:52.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.144+0000 7ff36b7fe640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff35406c920 0x7ff35406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.144+0000 7ff36b7fe640 1 -- 172.21.15.12:0/1877227569 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7ff36c08b230 con 0x7ff3740cf410 2024-04-15T15:21:52.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.145+0000 7ff372ffd640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff35406c920 0x7ff35406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.145+0000 7ff372ffd640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff35406c920 0x7ff35406ede0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff364035000 tx=0x7ff364030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.149+0000 7ff36b7fe640 1 -- 172.21.15.12:0/1877227569 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff37414ff00 con 0x7ff3740cf410 2024-04-15T15:21:52.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.335+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3740cecb0 con 0x7ff3740cf410 2024-04-15T15:21:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.336+0000 7ff36b7fe640 1 -- 172.21.15.12:0/1877227569 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7ff36c058bd0 con 0x7ff3740cf410 2024-04-15T15:21:52.338 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:52.338 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff35406c920 msgr2=0x7ff35406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff35406c920 0x7ff35406ede0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff364035000 tx=0x7ff364030000 comp rx=0 tx=0).stop 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3740cf410 msgr2=0x7ff37406caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3740cf410 0x7ff37406caf0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00c390 tx=0x7ff36c00c850 comp rx=0 tx=0).stop 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 shutdown_connections 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3740cf410 0x7ff37406caf0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff35406c920 0x7ff35406ede0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff37409b650 0x7ff3740ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 --2- 172.21.15.12:0/1877227569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff37409af10 0x7ff37414f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 >> 172.21.15.12:0/1877227569 conn(0x7ff3740b3fb0 msgr2=0x7ff3740d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 shutdown_connections 2024-04-15T15:21:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.338+0000 7ff379731640 1 -- 172.21.15.12:0/1877227569 wait complete. 2024-04-15T15:21:52.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:52.351 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:52.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.458+0000 7fda30e37640 1 Processor -- start 2024-04-15T15:21:52.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.458+0000 7fda30e37640 1 -- start start 2024-04-15T15:21:52.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.458+0000 7fda30e37640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c072d80 0x7fda2c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda30e37640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fda2c0af3e0 0x7fda2c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda30e37640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda2c15db40 con 0x7fda2c0d0690 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda30e37640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fda2c15cfc0 con 0x7fda2c0af3e0 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda30e37640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fda2c15d2a0 con 0x7fda2c072d80 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda23fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fda2c0af3e0 0x7fda2c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda2a59c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c072d80 0x7fda2c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda2ad9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fda2c0d0690 0x7fda2c06e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:56056/0 (socket says 172.21.15.12:56056) 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda2ad9d640 1 -- 172.21.15.12:0/1842347266 learned_addr learned my addr 172.21.15.12:0/1842347266 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda2a59c640 1 -- 172.21.15.12:0/1842347266 >> v2:172.21.15.55:3300/0 conn(0x7fda2c0af3e0 msgr2=0x7fda2c0af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda2a59c640 1 --2- 172.21.15.12:0/1842347266 >> v2:172.21.15.55:3300/0 conn(0x7fda2c0af3e0 0x7fda2c0af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda29d9b640 1 -- 172.21.15.12:0/1842347266 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4076141132 0 0) 0x7fda2c15db40 con 0x7fda2c0d0690 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda2a59c640 1 -- 172.21.15.12:0/1842347266 >> v1:172.21.15.55:6789/0 conn(0x7fda2c0d0690 legacy=0x7fda2c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.459+0000 7fda2a59c640 1 -- 172.21.15.12:0/1842347266 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c15d580 con 0x7fda2c072d80 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.460+0000 7fda2a59c640 1 --2- 172.21.15.12:0/1842347266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c072d80 0x7fda2c071d10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0097a0 tx=0x7fda1c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=350bcd3bdf6fc10e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.460+0000 7fda29d9b640 1 -- 172.21.15.12:0/1842347266 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0315a0 con 0x7fda2c072d80 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.460+0000 7fda29d9b640 1 -- 172.21.15.12:0/1842347266 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda1c037070 con 0x7fda2c072d80 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.460+0000 7fda29d9b640 1 -- 172.21.15.12:0/1842347266 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c031960 con 0x7fda2c072d80 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.461+0000 7fda30e37640 1 -- 172.21.15.12:0/1842347266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c072d80 msgr2=0x7fda2c071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.461+0000 7fda30e37640 1 --2- 172.21.15.12:0/1842347266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c072d80 0x7fda2c071d10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0097a0 tx=0x7fda1c02b840 comp rx=0 tx=0).stop 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.461+0000 7fda30e37640 1 -- 172.21.15.12:0/1842347266 shutdown_connections 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.461+0000 7fda30e37640 1 --2- 172.21.15.12:0/1842347266 >> v2:172.21.15.55:3300/0 conn(0x7fda2c0af3e0 0x7fda2c0af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.461+0000 7fda30e37640 1 --2- 172.21.15.12:0/1842347266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c072d80 0x7fda2c071d10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.461+0000 7fda30e37640 1 -- 172.21.15.12:0/1842347266 >> 172.21.15.12:0/1842347266 conn(0x7fda2c06be00 msgr2=0x7fda2c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:52.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.461+0000 7fda30e37640 1 -- 172.21.15.12:0/1842347266 shutdown_connections 2024-04-15T15:21:52.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.461+0000 7fda30e37640 1 -- 172.21.15.12:0/1842347266 wait complete. 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.462+0000 7fda30e37640 1 Processor -- start 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.462+0000 7fda30e37640 1 -- start start 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda30e37640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda2c072d80 0x7fda2c14c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda30e37640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c0af3e0 0x7fda2c147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda30e37640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda2c0d0690 0x7fda2c154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda30e37640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c15cd10 con 0x7fda2c0d0690 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda30e37640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c15cb90 con 0x7fda2c072d80 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda30e37640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fda2c0d0b20 con 0x7fda2c0af3e0 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda2c0d0690 0x7fda2c154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda2c072d80 0x7fda2c14c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda2c072d80 0x7fda2c14c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:43376/0 (socket says 172.21.15.12:43376) 2024-04-15T15:21:52.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2a59c640 1 -- 172.21.15.12:0/2120504068 learned_addr learned my addr 172.21.15.12:0/2120504068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda23fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c0af3e0 0x7fda2c147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2a59c640 1 -- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c0af3e0 msgr2=0x7fda2c147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2a59c640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c0af3e0 0x7fda2c147680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2a59c640 1 -- 172.21.15.12:0/2120504068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda2c0d0690 msgr2=0x7fda2c154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2a59c640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda2c0d0690 0x7fda2c154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2a59c640 1 -- 172.21.15.12:0/2120504068 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c0b41e0 con 0x7fda2c072d80 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.463+0000 7fda2ad9d640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda2c0d0690 0x7fda2c154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.464+0000 7fda2a59c640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda2c072d80 0x7fda2c14c030 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fda1c034040 tx=0x7fda1c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.464+0000 7fda22ffd640 1 -- 172.21.15.12:0/2120504068 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c032630 con 0x7fda2c072d80 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.464+0000 7fda22ffd640 1 -- 172.21.15.12:0/2120504068 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda1c0327f0 con 0x7fda2c072d80 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.464+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda2c14b550 con 0x7fda2c072d80 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.464+0000 7fda22ffd640 1 -- 172.21.15.12:0/2120504068 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c032be0 con 0x7fda2c072d80 2024-04-15T15:21:52.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.464+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda2c14bb10 con 0x7fda2c072d80 2024-04-15T15:21:52.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.465+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda2c0689e0 con 0x7fda2c072d80 2024-04-15T15:21:52.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.465+0000 7fda22ffd640 1 -- 172.21.15.12:0/2120504068 <== mon.1 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda1c03b3f0 con 0x7fda2c072d80 2024-04-15T15:21:52.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.466+0000 7fda22ffd640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda0c06c8f0 0x7fda0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.466+0000 7fda22ffd640 1 -- 172.21.15.12:0/2120504068 <== mon.1 v2:172.21.15.55:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fda1c02f030 con 0x7fda2c072d80 2024-04-15T15:21:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.466+0000 7fda23fff640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda0c06c8f0 0x7fda0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.466+0000 7fda23fff640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda0c06c8f0 0x7fda0c06edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fda2c15c8c0 tx=0x7fda14009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.471+0000 7fda22ffd640 1 -- 172.21.15.12:0/2120504068 <== mon.1 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda1c040070 con 0x7fda2c072d80 2024-04-15T15:21:52.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.657+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fda2c072170 con 0x7fda2c072d80 2024-04-15T15:21:52.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.658+0000 7fda22ffd640 1 -- 172.21.15.12:0/2120504068 <== mon.1 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fda1c07b610 con 0x7fda2c072d80 2024-04-15T15:21:52.659 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda0c06c8f0 msgr2=0x7fda0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda0c06c8f0 0x7fda0c06edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fda2c15c8c0 tx=0x7fda14009190 comp rx=0 tx=0).stop 2024-04-15T15:21:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda2c072d80 msgr2=0x7fda2c14c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda2c072d80 0x7fda2c14c030 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fda1c034040 tx=0x7fda1c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 shutdown_connections 2024-04-15T15:21:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda2c0d0690 0x7fda2c154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda2c0af3e0 0x7fda2c147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda0c06c8f0 0x7fda0c06edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.659+0000 7fda30e37640 1 --2- 172.21.15.12:0/2120504068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda2c072d80 0x7fda2c14c030 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.660+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 >> 172.21.15.12:0/2120504068 conn(0x7fda2c06be00 msgr2=0x7fda2c0b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.660+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 shutdown_connections 2024-04-15T15:21:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.660+0000 7fda30e37640 1 -- 172.21.15.12:0/2120504068 wait complete. 2024-04-15T15:21:52.672 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.778+0000 7f6fdaee4640 1 Processor -- start 2024-04-15T15:21:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.778+0000 7f6fdaee4640 1 -- start start 2024-04-15T15:21:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fdaee4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6fd415e790 0x7f6fd40b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fdaee4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40b7be0 0x7f6fd40d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fdaee4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd415eb70 con 0x7f6fd40d1b20 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fdaee4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6fd40d3790 con 0x7f6fd415e790 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fdaee4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd40d45b0 con 0x7f6fd40b7be0 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fd8c80640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6fd415e790 0x7f6fd40b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fd9481640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6fd40d1b20 0x7f6fd40d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33468/0 (socket says 172.21.15.12:33468) 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fd9481640 1 -- 172.21.15.12:0/829344669 learned_addr learned my addr 172.21.15.12:0/829344669 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fd8c80640 1 -- 172.21.15.12:0/829344669 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40b7be0 msgr2=0x7f6fd40d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fd8c80640 1 --2- 172.21.15.12:0/829344669 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40b7be0 0x7f6fd40d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fd8c80640 1 -- 172.21.15.12:0/829344669 >> v1:172.21.15.12:6789/0 conn(0x7f6fd40d1b20 legacy=0x7f6fd40d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fd37fe640 1 -- 172.21.15.12:0/829344669 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2642013877 0 0) 0x7f6fd415eb70 con 0x7f6fd40d1b20 2024-04-15T15:21:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.779+0000 7f6fd8c80640 1 -- 172.21.15.12:0/829344669 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd40d3c80 con 0x7f6fd415e790 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fd8c80640 1 --2- 172.21.15.12:0/829344669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd415e790 0x7f6fd40b76a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80097a0 tx=0x7f6fc802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d11750455f531c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fd37fe640 1 -- 172.21.15.12:0/829344669 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8038070 con 0x7f6fd415e790 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fd37fe640 1 -- 172.21.15.12:0/829344669 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fc8031700 con 0x7f6fd415e790 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fd37fe640 1 -- 172.21.15.12:0/829344669 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8031ad0 con 0x7f6fd415e790 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fdaee4640 1 -- 172.21.15.12:0/829344669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd415e790 msgr2=0x7f6fd40b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/829344669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd415e790 0x7f6fd40b76a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80097a0 tx=0x7f6fc802b7a0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fdaee4640 1 -- 172.21.15.12:0/829344669 shutdown_connections 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/829344669 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40b7be0 0x7f6fd40d15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/829344669 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd415e790 0x7f6fd40b76a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.780+0000 7f6fdaee4640 1 -- 172.21.15.12:0/829344669 >> 172.21.15.12:0/829344669 conn(0x7f6fd4067b40 msgr2=0x7f6fd4067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.781+0000 7f6fdaee4640 1 -- 172.21.15.12:0/829344669 shutdown_connections 2024-04-15T15:21:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.781+0000 7f6fdaee4640 1 -- 172.21.15.12:0/829344669 wait complete. 2024-04-15T15:21:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.782+0000 7f6fdaee4640 1 Processor -- start 2024-04-15T15:21:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.782+0000 7f6fdaee4640 1 -- start start 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.782+0000 7f6fdaee4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd40b7be0 0x7f6fd41f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.782+0000 7f6fdaee4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40d1b20 0x7f6fd4149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fdaee4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fd415e790 0x7f6fd40b8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fdaee4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd40d5150 con 0x7f6fd40b7be0 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fdaee4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd40d4240 con 0x7f6fd415e790 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fdaee4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd40d52d0 con 0x7f6fd40d1b20 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd40b7be0 0x7f6fd41f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd40b7be0 0x7f6fd41f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52516/0 (socket says 172.21.15.12:52516) 2024-04-15T15:21:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 -- 172.21.15.12:0/3309789704 learned_addr learned my addr 172.21.15.12:0/3309789704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd9481640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fd415e790 0x7f6fd40b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd3fff640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40d1b20 0x7f6fd4149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 -- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40d1b20 msgr2=0x7f6fd4149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40d1b20 0x7f6fd4149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 -- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fd415e790 msgr2=0x7f6fd40b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fd415e790 0x7f6fd40b8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 -- 172.21.15.12:0/3309789704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd40c9770 con 0x7f6fd40b7be0 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd9481640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fd415e790 0x7f6fd40b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd3fff640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40d1b20 0x7f6fd4149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd8c80640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd40b7be0 0x7f6fd41f3e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80097a0 tx=0x7f6fc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd17fa640 1 -- 172.21.15.12:0/3309789704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8043070 con 0x7f6fd40b7be0 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd17fa640 1 -- 172.21.15.12:0/3309789704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fc8033a30 con 0x7f6fd40b7be0 2024-04-15T15:21:52.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd40c9a50 con 0x7f6fd40b7be0 2024-04-15T15:21:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fd17fa640 1 -- 172.21.15.12:0/3309789704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8038040 con 0x7f6fd40b7be0 2024-04-15T15:21:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.783+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd40ca010 con 0x7f6fd40b7be0 2024-04-15T15:21:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.784+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fd40689e0 con 0x7f6fd40b7be0 2024-04-15T15:21:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.785+0000 7f6fd17fa640 1 -- 172.21.15.12:0/3309789704 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6fc8046070 con 0x7f6fd40b7be0 2024-04-15T15:21:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.785+0000 7f6fd17fa640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6fb806c8f0 0x7f6fb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.785+0000 7f6fd17fa640 1 -- 172.21.15.12:0/3309789704 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f6fc80add00 con 0x7f6fd40b7be0 2024-04-15T15:21:52.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.785+0000 7f6fd3fff640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6fb806c8f0 0x7f6fb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:52.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.786+0000 7f6fd3fff640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6fb806c8f0 0x7f6fb806edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006fd0 tx=0x7f6fc0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.790+0000 7f6fd17fa640 1 -- 172.21.15.12:0/3309789704 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6fc8030070 con 0x7f6fd40b7be0 2024-04-15T15:21:52.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.979+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6fd40ba060 con 0x7f6fd40b7be0 2024-04-15T15:21:52.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.979+0000 7f6fd17fa640 1 -- 172.21.15.12:0/3309789704 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f6fc807bea0 con 0x7f6fd40b7be0 2024-04-15T15:21:52.981 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:52.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6fb806c8f0 msgr2=0x7f6fb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6fb806c8f0 0x7f6fb806edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006fd0 tx=0x7f6fc0008040 comp rx=0 tx=0).stop 2024-04-15T15:21:52.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd40b7be0 msgr2=0x7f6fd41f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:52.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd40b7be0 0x7f6fd41f3e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80097a0 tx=0x7f6fc80046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 shutdown_connections 2024-04-15T15:21:52.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6fb806c8f0 0x7f6fb806edb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fd415e790 0x7f6fd40b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6fd40d1b20 0x7f6fd4149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 --2- 172.21.15.12:0/3309789704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fd40b7be0 0x7f6fd41f3e80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:52.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 >> 172.21.15.12:0/3309789704 conn(0x7f6fd4067b40 msgr2=0x7f6fd40d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:52.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 shutdown_connections 2024-04-15T15:21:52.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:52.981+0000 7f6fdaee4640 1 -- 172.21.15.12:0/3309789704 wait complete. 2024-04-15T15:21:52.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.100+0000 7f6f7b5f0640 1 Processor -- start 2024-04-15T15:21:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.100+0000 7f6f7b5f0640 1 -- start start 2024-04-15T15:21:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f7b5f0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f7409b880 0x7f6f74070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f7b5f0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6f740d0da0 0x7f6f740d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f7b5f0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f740d05e0 con 0x7f6f74070940 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f7b5f0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6f7409bce0 con 0x7f6f7409b880 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f7b5f0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6f74071260 con 0x7f6f740d0da0 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f78b8b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f7409b880 0x7f6f74070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f7938c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6f74070940 0x7f6f74160310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33470/0 (socket says 172.21.15.12:33470) 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f7938c640 1 -- 172.21.15.12:0/42346461 learned_addr learned my addr 172.21.15.12:0/42346461 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f79b8d640 1 --2- 172.21.15.12:0/42346461 >> v2:172.21.15.55:3300/0 conn(0x7f6f740d0da0 0x7f6f740d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f78b8b640 1 -- 172.21.15.12:0/42346461 >> v2:172.21.15.55:3300/0 conn(0x7f6f740d0da0 msgr2=0x7f6f740d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f78b8b640 1 --2- 172.21.15.12:0/42346461 >> v2:172.21.15.55:3300/0 conn(0x7f6f740d0da0 0x7f6f740d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f78b8b640 1 -- 172.21.15.12:0/42346461 >> v1:172.21.15.12:6789/0 conn(0x7f6f74070940 legacy=0x7f6f74160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.101+0000 7f6f78b8b640 1 -- 172.21.15.12:0/42346461 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7415f720 con 0x7f6f7409b880 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.102+0000 7f6f78b8b640 1 --2- 172.21.15.12:0/42346461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7409b880 0x7f6f74070d20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6f74072160 tx=0x7f6f6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c019b40cae9301c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.102+0000 7f6f6bfff640 1 -- 172.21.15.12:0/42346461 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64035020 con 0x7f6f7409b880 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.102+0000 7f6f6bfff640 1 -- 172.21.15.12:0/42346461 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f64031830 con 0x7f6f7409b880 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.102+0000 7f6f6bfff640 1 -- 172.21.15.12:0/42346461 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64031c00 con 0x7f6f7409b880 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.103+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/42346461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7409b880 msgr2=0x7f6f74070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.103+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/42346461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7409b880 0x7f6f74070d20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6f74072160 tx=0x7f6f6402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.103+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/42346461 shutdown_connections 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.103+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/42346461 >> v2:172.21.15.55:3300/0 conn(0x7f6f740d0da0 0x7f6f740d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.103+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/42346461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f7409b880 0x7f6f74070d20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.103+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/42346461 >> 172.21.15.12:0/42346461 conn(0x7f6f740b3f40 msgr2=0x7f6f740b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:53.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.103+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/42346461 shutdown_connections 2024-04-15T15:21:53.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.103+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/42346461 wait complete. 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.104+0000 7f6f7b5f0640 1 Processor -- start 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.104+0000 7f6f7b5f0640 1 -- start start 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f7b5f0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f74070940 0x7f6f740b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f7b5f0640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f7409b880 0x7f6f740c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f7b5f0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f740d0da0 0x7f6f7415ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f7b5f0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f740d0880 con 0x7f6f740d0da0 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f7b5f0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7415fce0 con 0x7f6f74070940 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f7b5f0640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6f740d1710 con 0x7f6f7409b880 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f740d0da0 0x7f6f7415ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f740d0da0 0x7f6f7415ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52524/0 (socket says 172.21.15.12:52524) 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 -- 172.21.15.12:0/3809834588 learned_addr learned my addr 172.21.15.12:0/3809834588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f7938c640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f74070940 0x7f6f740b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f78b8b640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f7409b880 0x7f6f740c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 -- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f7409b880 msgr2=0x7f6f740c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f7409b880 0x7f6f740c7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 -- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f74070940 msgr2=0x7f6f740b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f74070940 0x7f6f740b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 -- 172.21.15.12:0/3809834588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7415d390 con 0x7f6f740d0da0 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f7938c640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f74070940 0x7f6f740b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f79b8d640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f740d0da0 0x7f6f7415ce50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c007e90 tx=0x7f6f6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.105+0000 7f6f69ffb640 1 -- 172.21.15.12:0/3809834588 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c014420 con 0x7f6f740d0da0 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.106+0000 7f6f69ffb640 1 -- 172.21.15.12:0/3809834588 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6c019070 con 0x7f6f740d0da0 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.106+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f740b6340 con 0x7f6f740d0da0 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.106+0000 7f6f69ffb640 1 -- 172.21.15.12:0/3809834588 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c014800 con 0x7f6f740d0da0 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.106+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f740b6900 con 0x7f6f740d0da0 2024-04-15T15:21:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.107+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f740689e0 con 0x7f6f740d0da0 2024-04-15T15:21:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.107+0000 7f6f69ffb640 1 -- 172.21.15.12:0/3809834588 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f6c0149a0 con 0x7f6f740d0da0 2024-04-15T15:21:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.107+0000 7f6f69ffb640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f5806c8f0 0x7f6f5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.108+0000 7f6f69ffb640 1 -- 172.21.15.12:0/3809834588 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f6f6c08c750 con 0x7f6f740d0da0 2024-04-15T15:21:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.108+0000 7f6f7938c640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f5806c8f0 0x7f6f5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.108+0000 7f6f7938c640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f5806c8f0 0x7f6f5806edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6f7409ad30 tx=0x7f6f70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.112+0000 7f6f69ffb640 1 -- 172.21.15.12:0/3809834588 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f7415d390 con 0x7f6f740d0da0 2024-04-15T15:21:53.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.299+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6f74070720 con 0x7f6f740d0da0 2024-04-15T15:21:53.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.299+0000 7f6f69ffb640 1 -- 172.21.15.12:0/3809834588 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f6f6c05a7d0 con 0x7f6f740d0da0 2024-04-15T15:21:53.300 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f5806c8f0 msgr2=0x7f6f5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f5806c8f0 0x7f6f5806edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6f7409ad30 tx=0x7f6f70009040 comp rx=0 tx=0).stop 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f740d0da0 msgr2=0x7f6f7415ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f740d0da0 0x7f6f7415ce50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c007e90 tx=0x7f6f6c00e4b0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 shutdown_connections 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f740d0da0 0x7f6f7415ce50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f7409b880 0x7f6f740c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f5806c8f0 0x7f6f5806edb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 --2- 172.21.15.12:0/3809834588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f74070940 0x7f6f740b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 >> 172.21.15.12:0/3809834588 conn(0x7f6f740b3f40 msgr2=0x7f6f741e6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 shutdown_connections 2024-04-15T15:21:53.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.301+0000 7f6f7b5f0640 1 -- 172.21.15.12:0/3809834588 wait complete. 2024-04-15T15:21:53.313 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.419+0000 7f2d2cc93640 1 Processor -- start 2024-04-15T15:21:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.420+0000 7f2d2cc93640 1 -- start start 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.420+0000 7f2d2cc93640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815e700 0x7f2d280b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.420+0000 7f2d2cc93640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d280d2360 0x7f2d280d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.420+0000 7f2d2cc93640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d280b47a0 con 0x7f2d2815eae0 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.420+0000 7f2d2cc93640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2d280b53b0 con 0x7f2d280d2360 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.420+0000 7f2d2cc93640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2d280b4a80 con 0x7f2d2815e700 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.420+0000 7f2d1ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d280d2360 0x7f2d280d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d26d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2d2815eae0 0x7f2d280cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33484/0 (socket says 172.21.15.12:33484) 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d26d9d640 1 -- 172.21.15.12:0/2709125654 learned_addr learned my addr 172.21.15.12:0/2709125654 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d2659c640 1 --2- 172.21.15.12:0/2709125654 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815e700 0x7f2d280b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d1ffff640 1 -- 172.21.15.12:0/2709125654 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815e700 msgr2=0x7f2d280b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d1ffff640 1 --2- 172.21.15.12:0/2709125654 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815e700 0x7f2d280b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d1ffff640 1 -- 172.21.15.12:0/2709125654 >> v1:172.21.15.12:6789/0 conn(0x7f2d2815eae0 legacy=0x7f2d280cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d25d9b640 1 -- 172.21.15.12:0/2709125654 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4091211149 0 0) 0x7f2d280b47a0 con 0x7f2d2815eae0 2024-04-15T15:21:53.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d1ffff640 1 -- 172.21.15.12:0/2709125654 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d280b4d60 con 0x7f2d280d2360 2024-04-15T15:21:53.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.421+0000 7f2d2659c640 1 --2- 172.21.15.12:0/2709125654 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815e700 0x7f2d280b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:53.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d1ffff640 1 --2- 172.21.15.12:0/2709125654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d280d2360 0x7f2d280d2b60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2d100097a0 tx=0x7f2d1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6422952d2456d4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d25d9b640 1 -- 172.21.15.12:0/2709125654 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10035020 con 0x7f2d280d2360 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d25d9b640 1 -- 172.21.15.12:0/2709125654 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d10031830 con 0x7f2d280d2360 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d25d9b640 1 -- 172.21.15.12:0/2709125654 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10031c00 con 0x7f2d280d2360 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d2cc93640 1 -- 172.21.15.12:0/2709125654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d280d2360 msgr2=0x7f2d280d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/2709125654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d280d2360 0x7f2d280d2b60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2d100097a0 tx=0x7f2d1002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d2cc93640 1 -- 172.21.15.12:0/2709125654 shutdown_connections 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/2709125654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d280d2360 0x7f2d280d2b60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/2709125654 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815e700 0x7f2d280b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d2cc93640 1 -- 172.21.15.12:0/2709125654 >> 172.21.15.12:0/2709125654 conn(0x7f2d2808bf60 msgr2=0x7f2d280d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d2cc93640 1 -- 172.21.15.12:0/2709125654 shutdown_connections 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.422+0000 7f2d2cc93640 1 -- 172.21.15.12:0/2709125654 wait complete. 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.424+0000 7f2d2cc93640 1 Processor -- start 2024-04-15T15:21:53.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.424+0000 7f2d2cc93640 1 -- start start 2024-04-15T15:21:53.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.424+0000 7f2d2cc93640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d280d2360 0x7f2d281606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.424+0000 7f2d2cc93640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2815e700 0x7f2d280cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.424+0000 7f2d2cc93640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815eae0 0x7f2d2815c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.424+0000 7f2d2cc93640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d280b5d80 con 0x7f2d2815e700 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.424+0000 7f2d2cc93640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2d280b4450 con 0x7f2d280d2360 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.424+0000 7f2d2cc93640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2d280d1d40 con 0x7f2d2815eae0 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2815e700 0x7f2d280cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2815e700 0x7f2d280cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52534/0 (socket says 172.21.15.12:52534) 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 -- 172.21.15.12:0/169706181 learned_addr learned my addr 172.21.15.12:0/169706181 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d2659c640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d280d2360 0x7f2d281606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d26d9d640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815eae0 0x7f2d2815c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 -- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815eae0 msgr2=0x7f2d2815c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815eae0 0x7f2d2815c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 -- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d280d2360 msgr2=0x7f2d281606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d280d2360 0x7f2d281606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 -- 172.21.15.12:0/169706181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d28160bf0 con 0x7f2d2815e700 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d2659c640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d280d2360 0x7f2d281606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1ffff640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2815e700 0x7f2d280cc9d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2d10000c00 tx=0x7f2d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1effd640 1 -- 172.21.15.12:0/169706181 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10035050 con 0x7f2d2815e700 2024-04-15T15:21:53.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1effd640 1 -- 172.21.15.12:0/169706181 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d10004230 con 0x7f2d2815e700 2024-04-15T15:21:53.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d280cb650 con 0x7f2d2815e700 2024-04-15T15:21:53.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d1effd640 1 -- 172.21.15.12:0/169706181 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1003b800 con 0x7f2d2815e700 2024-04-15T15:21:53.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.425+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d280cbc10 con 0x7f2d2815e700 2024-04-15T15:21:53.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.426+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d280689e0 con 0x7f2d2815e700 2024-04-15T15:21:53.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.427+0000 7f2d1effd640 1 -- 172.21.15.12:0/169706181 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d10038020 con 0x7f2d2815e700 2024-04-15T15:21:53.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.427+0000 7f2d1effd640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2d0806c920 0x7f2d0806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.428+0000 7f2d1effd640 1 -- 172.21.15.12:0/169706181 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f2d100add10 con 0x7f2d2815e700 2024-04-15T15:21:53.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.428+0000 7f2d2659c640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2d0806c920 0x7f2d0806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.428+0000 7f2d2659c640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2d0806c920 0x7f2d0806ede0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2d1800aa20 tx=0x7f2d18008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.432+0000 7f2d1effd640 1 -- 172.21.15.12:0/169706181 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d10040070 con 0x7f2d2815e700 2024-04-15T15:21:53.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.620+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2d280d1070 con 0x7f2d2815e700 2024-04-15T15:21:53.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.620+0000 7f2d1effd640 1 -- 172.21.15.12:0/169706181 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f2d1007bd90 con 0x7f2d2815e700 2024-04-15T15:21:53.621 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:53.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2d0806c920 msgr2=0x7f2d0806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2d0806c920 0x7f2d0806ede0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2d1800aa20 tx=0x7f2d18008040 comp rx=0 tx=0).stop 2024-04-15T15:21:53.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2815e700 msgr2=0x7f2d280cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2815e700 0x7f2d280cc9d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2d10000c00 tx=0x7f2d100046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 shutdown_connections 2024-04-15T15:21:53.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2d2815eae0 0x7f2d2815c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d2815e700 0x7f2d280cc9d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2d0806c920 0x7f2d0806ede0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 --2- 172.21.15.12:0/169706181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d280d2360 0x7f2d281606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 >> 172.21.15.12:0/169706181 conn(0x7f2d2808bf60 msgr2=0x7f2d280cfad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:53.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 shutdown_connections 2024-04-15T15:21:53.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.622+0000 7f2d2cc93640 1 -- 172.21.15.12:0/169706181 wait complete. 2024-04-15T15:21:53.635 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:53.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.738+0000 7f3641d67640 1 Processor -- start 2024-04-15T15:21:53.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.738+0000 7f3641d67640 1 -- start start 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f3641d67640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f363c0d5150 0x7f363c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f3641d67640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f363c15eb70 con 0x7f363c0d15e0 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f3641d67640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f363c0d3c80 con 0x7f363c15e790 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f3641d67640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f363c0d3790 con 0x7f363c0d5150 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f363bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f363c0d5150 0x7f363c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f363affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f363c0d15e0 0x7f363c0d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33498/0 (socket says 172.21.15.12:33498) 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f363affd640 1 -- 172.21.15.12:0/3914158684 learned_addr learned my addr 172.21.15.12:0/3914158684 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f363bfff640 1 -- 172.21.15.12:0/3914158684 >> v1:172.21.15.55:6789/0 conn(0x7f363c15e790 legacy=0x7f363c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f363bfff640 1 -- 172.21.15.12:0/3914158684 >> v1:172.21.15.12:6789/0 conn(0x7f363c0d15e0 legacy=0x7f363c0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f363a7fc640 1 -- 172.21.15.12:0/3914158684 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374188656 0 0) 0x7f363c15eb70 con 0x7f363c0d15e0 2024-04-15T15:21:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.739+0000 7f363bfff640 1 -- 172.21.15.12:0/3914158684 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363c0d3f60 con 0x7f363c0d5150 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.740+0000 7f363bfff640 1 --2- 172.21.15.12:0/3914158684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0d1a90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3634009d90 tx=0x7f363402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec57748791ecca3b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.740+0000 7f363a7fc640 1 -- 172.21.15.12:0/3914158684 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3634039070 con 0x7f363c0d5150 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.740+0000 7f363a7fc640 1 -- 172.21.15.12:0/3914158684 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f363402cbd0 con 0x7f363c0d5150 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.740+0000 7f363a7fc640 1 -- 172.21.15.12:0/3914158684 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3634034440 con 0x7f363c0d5150 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.741+0000 7f3641d67640 1 -- 172.21.15.12:0/3914158684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 msgr2=0x7f363c0d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.741+0000 7f3641d67640 1 --2- 172.21.15.12:0/3914158684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0d1a90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3634009d90 tx=0x7f363402be20 comp rx=0 tx=0).stop 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.741+0000 7f3641d67640 1 -- 172.21.15.12:0/3914158684 shutdown_connections 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.741+0000 7f3641d67640 1 --2- 172.21.15.12:0/3914158684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0d1a90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.741+0000 7f3641d67640 1 -- 172.21.15.12:0/3914158684 >> 172.21.15.12:0/3914158684 conn(0x7f363c067b40 msgr2=0x7f363c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.741+0000 7f3641d67640 1 -- 172.21.15.12:0/3914158684 shutdown_connections 2024-04-15T15:21:53.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.741+0000 7f3641d67640 1 -- 172.21.15.12:0/3914158684 wait complete. 2024-04-15T15:21:53.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.742+0000 7f3641d67640 1 Processor -- start 2024-04-15T15:21:53.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.742+0000 7f3641d67640 1 -- start start 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f3641d67640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f363c0d15e0 0x7f363c0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f3641d67640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f3641d67640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f363c15e790 0x7f363c0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f3641d67640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f363c0b7db0 con 0x7f363c0d5150 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f3641d67640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f363c0d45b0 con 0x7f363c0d15e0 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f3641d67640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f363c0d29c0 con 0x7f363c15e790 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52560/0 (socket says 172.21.15.12:52560) 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 -- 172.21.15.12:0/503904255 learned_addr learned my addr 172.21.15.12:0/503904255 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363b7fe640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f363c0d15e0 0x7f363c0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 -- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f363c15e790 msgr2=0x7f363c0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f363c15e790 0x7f363c0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 -- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f363c0d15e0 msgr2=0x7f363c0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f363c0d15e0 0x7f363c0c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 -- 172.21.15.12:0/503904255 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363c0c76a0 con 0x7f363c0d5150 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363b7fe640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f363c0d15e0 0x7f363c0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.743+0000 7f363affd640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0cc600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f363000c4a0 tx=0x7f363000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.744+0000 7f3623fff640 1 -- 172.21.15.12:0/503904255 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630018070 con 0x7f363c0d5150 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.744+0000 7f3623fff640 1 -- 172.21.15.12:0/503904255 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36300074e0 con 0x7f363c0d5150 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.744+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f363c0cd080 con 0x7f363c0d5150 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.744+0000 7f3623fff640 1 -- 172.21.15.12:0/503904255 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630004a20 con 0x7f363c0d5150 2024-04-15T15:21:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.744+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f363c0cd8f0 con 0x7f363c0d5150 2024-04-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.745+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f363c0689e0 con 0x7f363c0d5150 2024-04-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.745+0000 7f3623fff640 1 -- 172.21.15.12:0/503904255 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f363001b070 con 0x7f363c0d5150 2024-04-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.746+0000 7f3623fff640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f362806c8f0 0x7f362806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.746+0000 7f3623fff640 1 -- 172.21.15.12:0/503904255 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f363008b210 con 0x7f363c0d5150 2024-04-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.746+0000 7f363b7fe640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f362806c8f0 0x7f362806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.746+0000 7f363b7fe640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f362806c8f0 0x7f362806edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f362c000f60 tx=0x7f362c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.750+0000 7f3623fff640 1 -- 172.21.15.12:0/503904255 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f363c0c76a0 con 0x7f363c0d5150 2024-04-15T15:21:53.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.935+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f363c0d2f40 con 0x7f363c0d5150 2024-04-15T15:21:53.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.937+0000 7f3623fff640 1 -- 172.21.15.12:0/503904255 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f3630059290 con 0x7f363c0d5150 2024-04-15T15:21:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:53.938 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:53.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f362806c8f0 msgr2=0x7f362806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f362806c8f0 0x7f362806edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f362c000f60 tx=0x7f362c009040 comp rx=0 tx=0).stop 2024-04-15T15:21:53.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 msgr2=0x7f363c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:53.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0cc600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f363000c4a0 tx=0x7f363000c960 comp rx=0 tx=0).stop 2024-04-15T15:21:53.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 shutdown_connections 2024-04-15T15:21:53.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f363c15e790 0x7f363c0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363c0d5150 0x7f363c0cc600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f362806c8f0 0x7f362806edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 --2- 172.21.15.12:0/503904255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f363c0d15e0 0x7f363c0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:53.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.938+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 >> 172.21.15.12:0/503904255 conn(0x7f363c067b40 msgr2=0x7f363c0b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:53.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.939+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 shutdown_connections 2024-04-15T15:21:53.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:53.939+0000 7f3641d67640 1 -- 172.21.15.12:0/503904255 wait complete. 2024-04-15T15:21:53.951 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:53.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.058+0000 7ff214f8d640 1 Processor -- start 2024-04-15T15:21:54.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.058+0000 7ff214f8d640 1 -- start start 2024-04-15T15:21:54.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff214f8d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff21015e790 0x7ff2100b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff214f8d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 0x7ff2100d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff214f8d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff21015eb70 con 0x7ff2100d1b20 2024-04-15T15:21:54.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff214f8d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff2100d3790 con 0x7ff21015e790 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff214f8d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff2100d45b0 con 0x7ff2100b7be0 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20ed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff2100d1b20 0x7ff2100d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33502/0 (socket says 172.21.15.12:33502) 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20ed9d640 1 -- 172.21.15.12:0/2645701515 learned_addr learned my addr 172.21.15.12:0/2645701515 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20e59c640 1 --2- 172.21.15.12:0/2645701515 >> v2:172.21.15.55:3300/0 conn(0x7ff21015e790 0x7ff2100b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff207fff640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 0x7ff2100d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 409634974 0 0) 0x7ff21015eb70 con 0x7ff2100d1b20 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1fc003520 con 0x7ff2100d1b20 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3964239382 0 0) 0x7ff1fc003520 con 0x7ff2100d1b20 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 msgr2=0x7ff2100d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 0x7ff2100d15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 >> v2:172.21.15.55:3300/0 conn(0x7ff21015e790 msgr2=0x7ff2100b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 --2- 172.21.15.12:0/2645701515 >> v2:172.21.15.55:3300/0 conn(0x7ff21015e790 0x7ff2100b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2100d3c80 con 0x7ff2100d1b20 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7ff208001a10 con 0x7ff2100d1b20 2024-04-15T15:21:54.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 >> v1:172.21.15.12:6789/0 conn(0x7ff2100d1b20 legacy=0x7ff2100d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.059+0000 7ff20dd9b640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 0x7ff1fc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20dd9b640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc003f30 0x7ff1fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20dd9b640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc004d40 0x7ff1fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100d3790 con 0x7ff1fc004d40 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100d45b0 con 0x7ff1fc003f30 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff1fc00baa0 con 0x7ff2100b7be0 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20ed9d640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc004d40 0x7ff1fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20e59c640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 0x7ff1fc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff207fff640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc003f30 0x7ff1fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20ed9d640 1 -- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 msgr2=0x7ff1fc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20ed9d640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 0x7ff1fc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20ed9d640 1 -- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc003f30 msgr2=0x7ff1fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20ed9d640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc003f30 0x7ff1fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20ed9d640 1 -- 172.21.15.12:0/2645701515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc003520 con 0x7ff1fc004d40 2024-04-15T15:21:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.060+0000 7ff20e59c640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff2100b7be0 0x7ff1fc0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff20ed9d640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc004d40 0x7ff1fc0051c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff20800a880 tx=0x7ff1fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62453cffd671645 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20800a4d0 con 0x7ff1fc004d40 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff208034630 con 0x7ff1fc004d40 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff20dd9b640 1 -- 172.21.15.12:0/2645701515 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff208034a60 con 0x7ff1fc004d40 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff214f8d640 1 -- 172.21.15.12:0/2645701515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc004d40 msgr2=0x7ff1fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff214f8d640 1 --2- 172.21.15.12:0/2645701515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1fc004d40 0x7ff1fc0051c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff20800a880 tx=0x7ff1fc005830 comp rx=0 tx=0).stop 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff20ed9d640 1 -- 172.21.15.12:0/2645701515 reap_dead start 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff214f8d640 1 -- 172.21.15.12:0/2645701515 shutdown_connections 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff214f8d640 1 -- 172.21.15.12:0/2645701515 >> 172.21.15.12:0/2645701515 conn(0x7ff210067b40 msgr2=0x7ff210067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff214f8d640 1 -- 172.21.15.12:0/2645701515 shutdown_connections 2024-04-15T15:21:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.061+0000 7ff214f8d640 1 -- 172.21.15.12:0/2645701515 wait complete. 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.063+0000 7ff214f8d640 1 Processor -- start 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.063+0000 7ff214f8d640 1 -- start start 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.063+0000 7ff214f8d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff1fc004d40 0x7ff210158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.063+0000 7ff214f8d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2100c8120 0x7ff2100cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.063+0000 7ff214f8d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2101595e0 0x7ff2100cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.063+0000 7ff214f8d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100d5150 con 0x7ff2100c8120 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.063+0000 7ff214f8d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100d4240 con 0x7ff2101595e0 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.063+0000 7ff214f8d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff2100d52d0 con 0x7ff1fc004d40 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2100c8120 0x7ff2100cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2100c8120 0x7ff2100cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52578/0 (socket says 172.21.15.12:52578) 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 -- 172.21.15.12:0/825117506 learned_addr learned my addr 172.21.15.12:0/825117506 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff20ed9d640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2101595e0 0x7ff2100cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 -- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff1fc004d40 msgr2=0x7ff210158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff20e59c640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff1fc004d40 0x7ff210158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff1fc004d40 0x7ff210158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 -- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2101595e0 msgr2=0x7ff2100cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2101595e0 0x7ff2100cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 -- 172.21.15.12:0/825117506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2100c85b0 con 0x7ff2100c8120 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff20ed9d640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2101595e0 0x7ff2100cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff20e59c640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff1fc004d40 0x7ff210158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff207fff640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2100c8120 0x7ff2100cd6f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8011540 tx=0x7ff1f8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff206ffd640 1 -- 172.21.15.12:0/825117506 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8015930 con 0x7ff2100c8120 2024-04-15T15:21:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff206ffd640 1 -- 172.21.15.12:0/825117506 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f8015ad0 con 0x7ff2100c8120 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff206ffd640 1 -- 172.21.15.12:0/825117506 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f800f460 con 0x7ff2100c8120 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2100c8890 con 0x7ff2100c8120 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.064+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2100c8e50 con 0x7ff2100c8120 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.065+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2100c9090 con 0x7ff2100c8120 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.066+0000 7ff206ffd640 1 -- 172.21.15.12:0/825117506 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1f800f600 con 0x7ff2100c8120 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.066+0000 7ff206ffd640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff1f006c8f0 0x7ff1f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.066+0000 7ff206ffd640 1 -- 172.21.15.12:0/825117506 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7ff1f808c870 con 0x7ff2100c8120 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.066+0000 7ff20e59c640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff1f006c8f0 0x7ff1f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.067+0000 7ff20e59c640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff1f006c8f0 0x7ff1f006edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff2100cec90 tx=0x7ff200008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.071+0000 7ff206ffd640 1 -- 172.21.15.12:0/825117506 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1f8019030 con 0x7ff2100c8120 2024-04-15T15:21:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.256+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff2100689e0 con 0x7ff2100c8120 2024-04-15T15:21:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.256+0000 7ff206ffd640 1 -- 172.21.15.12:0/825117506 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7ff1f805a8f0 con 0x7ff2100c8120 2024-04-15T15:21:54.257 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff1f006c8f0 msgr2=0x7ff1f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff1f006c8f0 0x7ff1f006edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff2100cec90 tx=0x7ff200008040 comp rx=0 tx=0).stop 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2100c8120 msgr2=0x7ff2100cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2100c8120 0x7ff2100cd6f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8011540 tx=0x7ff1f8011a00 comp rx=0 tx=0).stop 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 shutdown_connections 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff1f006c8f0 0x7ff1f006edb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2101595e0 0x7ff2100cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2100c8120 0x7ff2100cd6f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 --2- 172.21.15.12:0/825117506 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff1fc004d40 0x7ff210158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 >> 172.21.15.12:0/825117506 conn(0x7ff210067b40 msgr2=0x7ff2100d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 shutdown_connections 2024-04-15T15:21:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.258+0000 7ff214f8d640 1 -- 172.21.15.12:0/825117506 wait complete. 2024-04-15T15:21:54.270 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:54.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.377+0000 7f5fb6552640 1 Processor -- start 2024-04-15T15:21:54.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.377+0000 7f5fb6552640 1 -- start start 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.378+0000 7f5fb6552640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5fb00d15c0 0x7f5fb006cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.378+0000 7f5fb6552640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb006dbd0 con 0x7f5fb015e7a0 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.378+0000 7f5fb6552640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb00d0530 con 0x7f5fb006d050 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.378+0000 7f5fb6552640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb00cf490 con 0x7f5fb00d15c0 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.378+0000 7f5faffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5fb015e7a0 0x7f5fb00d0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33508/0 (socket says 172.21.15.12:33508) 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.378+0000 7f5faffff640 1 -- 172.21.15.12:0/1274609040 learned_addr learned my addr 172.21.15.12:0/1274609040 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.379+0000 7f5fa7fff640 1 --2- 172.21.15.12:0/1274609040 >> v2:172.21.15.55:3300/0 conn(0x7f5fb00d15c0 0x7f5fb006cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.379+0000 7f5faf7fe640 1 -- 172.21.15.12:0/1274609040 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2763172866 0 0) 0x7f5fa0001110 con 0x7f5fb015e7a0 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.379+0000 7f5faf7fe640 1 -- 172.21.15.12:0/1274609040 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f9c003520 con 0x7f5fb015e7a0 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.379+0000 7f5fa7fff640 1 -- 172.21.15.12:0/1274609040 >> v1:172.21.15.55:6789/0 conn(0x7f5fb006d050 legacy=0x7f5fb006d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.379+0000 7f5fa7fff640 1 -- 172.21.15.12:0/1274609040 >> v1:172.21.15.12:6789/0 conn(0x7f5fb015e7a0 legacy=0x7f5fb00d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.379+0000 7f5fa7fff640 1 -- 172.21.15.12:0/1274609040 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb00cf770 con 0x7f5fb00d15c0 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.380+0000 7f5fa7fff640 1 --2- 172.21.15.12:0/1274609040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb00d15c0 0x7f5fb006cb10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5f980097a0 tx=0x7f5f9802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7e707b2e1bb9c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.380+0000 7f5faf7fe640 1 -- 172.21.15.12:0/1274609040 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98035020 con 0x7f5fb00d15c0 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.380+0000 7f5faf7fe640 1 -- 172.21.15.12:0/1274609040 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f98031830 con 0x7f5fb00d15c0 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.380+0000 7f5faf7fe640 1 -- 172.21.15.12:0/1274609040 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98031be0 con 0x7f5fb00d15c0 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.381+0000 7f5fb6552640 1 -- 172.21.15.12:0/1274609040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb00d15c0 msgr2=0x7f5fb006cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.381+0000 7f5fb6552640 1 --2- 172.21.15.12:0/1274609040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb00d15c0 0x7f5fb006cb10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5f980097a0 tx=0x7f5f9802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.381+0000 7f5fb6552640 1 -- 172.21.15.12:0/1274609040 shutdown_connections 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.381+0000 7f5fb6552640 1 --2- 172.21.15.12:0/1274609040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb00d15c0 0x7f5fb006cb10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.381+0000 7f5fb6552640 1 -- 172.21.15.12:0/1274609040 >> 172.21.15.12:0/1274609040 conn(0x7f5fb008bf60 msgr2=0x7f5fb00d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:54.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.381+0000 7f5fb6552640 1 -- 172.21.15.12:0/1274609040 shutdown_connections 2024-04-15T15:21:54.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.381+0000 7f5fb6552640 1 -- 172.21.15.12:0/1274609040 wait complete. 2024-04-15T15:21:54.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.382+0000 7f5fb6552640 1 Processor -- start 2024-04-15T15:21:54.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.382+0000 7f5fb6552640 1 -- start start 2024-04-15T15:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb6552640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fb006d050 0x7f5fb00c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb6552640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb00d15c0 0x7f5fb015fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb6552640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb015e7a0 0x7f5fb01603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb6552640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb00cf180 con 0x7f5fb00d15c0 2024-04-15T15:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb6552640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb00cfd30 con 0x7f5fb015e7a0 2024-04-15T15:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb6552640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb015eb80 con 0x7f5fb006d050 2024-04-15T15:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb00d15c0 0x7f5fb015fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb00d15c0 0x7f5fb015fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52580/0 (socket says 172.21.15.12:52580) 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 -- 172.21.15.12:0/3835081042 learned_addr learned my addr 172.21.15.12:0/3835081042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb4aef640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb015e7a0 0x7f5fb01603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5faffff640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fb006d050 0x7f5fb00c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 -- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fb006d050 msgr2=0x7f5fb00c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fb006d050 0x7f5fb00c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 -- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb015e7a0 msgr2=0x7f5fb01603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb015e7a0 0x7f5fb01603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 -- 172.21.15.12:0/3835081042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb00c8eb0 con 0x7f5fb00d15c0 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb4aef640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb015e7a0 0x7f5fb01603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fa7fff640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb00d15c0 0x7f5fb015fe70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5f980097a0 tx=0x7f5f980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fad7fa640 1 -- 172.21.15.12:0/3835081042 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98048070 con 0x7f5fb00d15c0 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fad7fa640 1 -- 172.21.15.12:0/3835081042 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f98004450 con 0x7f5fb00d15c0 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb00c9190 con 0x7f5fb00d15c0 2024-04-15T15:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.383+0000 7f5fad7fa640 1 -- 172.21.15.12:0/3835081042 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98035070 con 0x7f5fb00d15c0 2024-04-15T15:21:54.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.384+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb00c9750 con 0x7f5fb00d15c0 2024-04-15T15:21:54.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.384+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb00689e0 con 0x7f5fb00d15c0 2024-04-15T15:21:54.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.385+0000 7f5fad7fa640 1 -- 172.21.15.12:0/3835081042 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f9804b070 con 0x7f5fb00d15c0 2024-04-15T15:21:54.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.385+0000 7f5fad7fa640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5f9006c8f0 0x7f5f9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.385+0000 7f5fad7fa640 1 -- 172.21.15.12:0/3835081042 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f5f980ad870 con 0x7f5fb00d15c0 2024-04-15T15:21:54.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.386+0000 7f5faffff640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5f9006c8f0 0x7f5f9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.386+0000 7f5faffff640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5f9006c8f0 0x7f5f9006edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0007fd0 tx=0x7f5fa0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.390+0000 7f5fad7fa640 1 -- 172.21.15.12:0/3835081042 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f98040070 con 0x7f5fb00d15c0 2024-04-15T15:21:54.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.572+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5fb00b2330 con 0x7f5fb00d15c0 2024-04-15T15:21:54.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.573+0000 7f5fad7fa640 1 -- 172.21.15.12:0/3835081042 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f5f9807b8f0 con 0x7f5fb00d15c0 2024-04-15T15:21:54.575 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5f9006c8f0 msgr2=0x7f5f9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5f9006c8f0 0x7f5f9006edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0007fd0 tx=0x7f5fa0009040 comp rx=0 tx=0).stop 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb00d15c0 msgr2=0x7f5fb015fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb00d15c0 0x7f5fb015fe70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5f980097a0 tx=0x7f5f980046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 shutdown_connections 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5f9006c8f0 0x7f5f9006edb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fb015e7a0 0x7f5fb01603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb00d15c0 0x7f5fb015fe70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 --2- 172.21.15.12:0/3835081042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5fb006d050 0x7f5fb00c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 >> 172.21.15.12:0/3835081042 conn(0x7f5fb008bf60 msgr2=0x7f5fb00b9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 shutdown_connections 2024-04-15T15:21:54.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.575+0000 7f5fb6552640 1 -- 172.21.15.12:0/3835081042 wait complete. 2024-04-15T15:21:54.587 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:54.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.693+0000 7f1fa2d49640 1 Processor -- start 2024-04-15T15:21:54.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.693+0000 7f1fa2d49640 1 -- start start 2024-04-15T15:21:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1fa2d49640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1f9c15e790 0x7f1f9c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1fa2d49640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f9c0cfd20 0x7f1f9c0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1fa2d49640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 0x7f1f9c0d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1fa2d49640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1f9c0d2db0 con 0x7f1f9c0cfd20 2024-04-15T15:21:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1fa2d49640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1f9c0d2480 con 0x7f1f9c15e790 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1fa2d49640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1f9c0d2760 con 0x7f1f9c0d0f00 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1f9bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f9c0cfd20 0x7f1f9c0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1f9bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f9c0cfd20 0x7f1f9c0d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52582/0 (socket says 172.21.15.12:52582) 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1f9bfff640 1 -- 172.21.15.12:0/1346344154 learned_addr learned my addr 172.21.15.12:0/1346344154 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1fa12e6640 1 --2- 172.21.15.12:0/1346344154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 0x7f1f9c0d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1fa0ae5640 1 --2- 172.21.15.12:0/1346344154 >> v2:172.21.15.55:3300/0 conn(0x7f1f9c15e790 0x7f1f9c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1f9bfff640 1 -- 172.21.15.12:0/1346344154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 msgr2=0x7f1f9c0d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1f9bfff640 1 --2- 172.21.15.12:0/1346344154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 0x7f1f9c0d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1f9bfff640 1 -- 172.21.15.12:0/1346344154 >> v2:172.21.15.55:3300/0 conn(0x7f1f9c15e790 msgr2=0x7f1f9c0d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1f9bfff640 1 --2- 172.21.15.12:0/1346344154 >> v2:172.21.15.55:3300/0 conn(0x7f1f9c15e790 0x7f1f9c0d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.694+0000 7f1f9bfff640 1 -- 172.21.15.12:0/1346344154 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f9c0d2a40 con 0x7f1f9c0cfd20 2024-04-15T15:21:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.695+0000 7f1f9bfff640 1 --2- 172.21.15.12:0/1346344154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c0cfd20 0x7f1f9c0d0160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0d4600 tx=0x7f1f9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2bb34a5d611a30b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.695+0000 7f1f9b7fe640 1 -- 172.21.15.12:0/1346344154 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90035020 con 0x7f1f9c0cfd20 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.695+0000 7f1f9b7fe640 1 -- 172.21.15.12:0/1346344154 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f90031870 con 0x7f1f9c0cfd20 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.695+0000 7f1f9b7fe640 1 -- 172.21.15.12:0/1346344154 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90031c40 con 0x7f1f9c0cfd20 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1346344154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c0cfd20 msgr2=0x7f1f9c0d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1346344154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c0cfd20 0x7f1f9c0d0160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0d4600 tx=0x7f1f9002b910 comp rx=0 tx=0).stop 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1346344154 shutdown_connections 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1346344154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c0cfd20 0x7f1f9c0d0160 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1346344154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 0x7f1f9c0d13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1346344154 >> v2:172.21.15.55:3300/0 conn(0x7f1f9c15e790 0x7f1f9c0d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1346344154 >> 172.21.15.12:0/1346344154 conn(0x7f1f9c08bf60 msgr2=0x7f1f9c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1346344154 shutdown_connections 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.696+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1346344154 wait complete. 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.697+0000 7f1fa2d49640 1 Processor -- start 2024-04-15T15:21:54.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.697+0000 7f1fa2d49640 1 -- start start 2024-04-15T15:21:54.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa2d49640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f9c0cfd20 0x7f1f9c0b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa2d49640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 0x7f1f9c0b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa2d49640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c15e790 0x7f1f9c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa2d49640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9c15ecf0 con 0x7f1f9c15e790 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa2d49640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9c15eb70 con 0x7f1f9c0cfd20 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa2d49640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1f9c0b74f0 con 0x7f1f9c0d0f00 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa12e6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c15e790 0x7f1f9c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa12e6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c15e790 0x7f1f9c157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52596/0 (socket says 172.21.15.12:52596) 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa12e6640 1 -- 172.21.15.12:0/1481967570 learned_addr learned my addr 172.21.15.12:0/1481967570 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1fa0ae5640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f9c0cfd20 0x7f1f9c0b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.698+0000 7f1f9bfff640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 0x7f1f9c0b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa12e6640 1 -- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 msgr2=0x7f1f9c0b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa12e6640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 0x7f1f9c0b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa12e6640 1 -- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f9c0cfd20 msgr2=0x7f1f9c0b9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa12e6640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f9c0cfd20 0x7f1f9c0b9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa12e6640 1 -- 172.21.15.12:0/1481967570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f9c0b2380 con 0x7f1f9c15e790 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa0ae5640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f9c0cfd20 0x7f1f9c0b9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:54.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa12e6640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c15e790 0x7f1f9c157520 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00d800 tx=0x7f1f8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1f997fa640 1 -- 172.21.15.12:0/1481967570 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c018020 con 0x7f1f9c15e790 2024-04-15T15:21:54.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1f997fa640 1 -- 172.21.15.12:0/1481967570 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f8c0096c0 con 0x7f1f9c15e790 2024-04-15T15:21:54.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1f997fa640 1 -- 172.21.15.12:0/1481967570 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c009a90 con 0x7f1f9c15e790 2024-04-15T15:21:54.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f9c0b2660 con 0x7f1f9c15e790 2024-04-15T15:21:54.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.699+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f9c0b2c20 con 0x7f1f9c15e790 2024-04-15T15:21:54.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.700+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f9c0689e0 con 0x7f1f9c15e790 2024-04-15T15:21:54.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.701+0000 7f1f997fa640 1 -- 172.21.15.12:0/1481967570 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f8c01b020 con 0x7f1f9c15e790 2024-04-15T15:21:54.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.701+0000 7f1f997fa640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f8806c920 0x7f1f8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:54.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.701+0000 7f1f997fa640 1 -- 172.21.15.12:0/1481967570 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f1f8c08c1f0 con 0x7f1f9c15e790 2024-04-15T15:21:54.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.701+0000 7f1fa0ae5640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f8806c920 0x7f1f8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:54.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.702+0000 7f1fa0ae5640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f8806c920 0x7f1f8806ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1f94009a80 tx=0x7f1f940039d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:54.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.705+0000 7f1f997fa640 1 -- 172.21.15.12:0/1481967570 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f8c022030 con 0x7f1f9c15e790 2024-04-15T15:21:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.890+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1f9c0d1ce0 con 0x7f1f9c15e790 2024-04-15T15:21:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.891+0000 7f1f997fa640 1 -- 172.21.15.12:0/1481967570 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f1f8c05a0d0 con 0x7f1f9c15e790 2024-04-15T15:21:54.892 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.892+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f8806c920 msgr2=0x7f1f8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.892+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f8806c920 0x7f1f8806ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1f94009a80 tx=0x7f1f940039d0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c15e790 msgr2=0x7f1f9c157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c15e790 0x7f1f9c157520 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c00d800 tx=0x7f1f8c00dcc0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 shutdown_connections 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f9c15e790 0x7f1f9c157520 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f9c0d0f00 0x7f1f9c0b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f8806c920 0x7f1f8806ede0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 --2- 172.21.15.12:0/1481967570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f9c0cfd20 0x7f1f9c0b9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 >> 172.21.15.12:0/1481967570 conn(0x7f1f9c08bf60 msgr2=0x7f1f9c0b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 shutdown_connections 2024-04-15T15:21:54.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:54.893+0000 7f1fa2d49640 1 -- 172.21.15.12:0/1481967570 wait complete. 2024-04-15T15:21:54.905 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:55.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.011+0000 7f008f172640 1 Processor -- start 2024-04-15T15:21:55.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.011+0000 7f008f172640 1 -- start start 2024-04-15T15:21:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008f172640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f008815e700 0x7f00880b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008f172640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f00880d3270 0x7f00880cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008f172640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00880b47a0 con 0x7f00880d2360 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008f172640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f00880b53b0 con 0x7f00880d3270 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008f172640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f00880b4a80 con 0x7f008815e700 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008d70f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f00880d3270 0x7f00880cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f0087fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f00880d2360 0x7f00880d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33520/0 (socket says 172.21.15.12:33520) 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f0087fff640 1 -- 172.21.15.12:0/1456838890 learned_addr learned my addr 172.21.15.12:0/1456838890 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008cf0e640 1 --2- 172.21.15.12:0/1456838890 >> v2:172.21.15.55:3300/0 conn(0x7f008815e700 0x7f00880b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008d70f640 1 -- 172.21.15.12:0/1456838890 >> v2:172.21.15.55:3300/0 conn(0x7f008815e700 msgr2=0x7f00880b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008d70f640 1 --2- 172.21.15.12:0/1456838890 >> v2:172.21.15.55:3300/0 conn(0x7f008815e700 0x7f00880b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008d70f640 1 -- 172.21.15.12:0/1456838890 >> v1:172.21.15.12:6789/0 conn(0x7f00880d2360 legacy=0x7f00880d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008d70f640 1 -- 172.21.15.12:0/1456838890 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00880b4d60 con 0x7f00880d3270 2024-04-15T15:21:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f00877fe640 1 -- 172.21.15.12:0/1456838890 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008449340 0 0) 0x7f00880b47a0 con 0x7f00880d2360 2024-04-15T15:21:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.012+0000 7f008cf0e640 1 --2- 172.21.15.12:0/1456838890 >> v2:172.21.15.55:3300/0 conn(0x7f008815e700 0x7f00880b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.013+0000 7f008d70f640 1 --2- 172.21.15.12:0/1456838890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d3270 0x7f00880cf370 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f008806c6c0 tx=0x7f007802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5232263f6e9e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.013+0000 7f00877fe640 1 -- 172.21.15.12:0/1456838890 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0078035020 con 0x7f00880d3270 2024-04-15T15:21:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.013+0000 7f00877fe640 1 -- 172.21.15.12:0/1456838890 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0078031830 con 0x7f00880d3270 2024-04-15T15:21:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.013+0000 7f00877fe640 1 -- 172.21.15.12:0/1456838890 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0078031c00 con 0x7f00880d3270 2024-04-15T15:21:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.014+0000 7f008f172640 1 -- 172.21.15.12:0/1456838890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d3270 msgr2=0x7f00880cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.014+0000 7f008f172640 1 --2- 172.21.15.12:0/1456838890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d3270 0x7f00880cf370 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f008806c6c0 tx=0x7f007802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.014+0000 7f008f172640 1 -- 172.21.15.12:0/1456838890 shutdown_connections 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.014+0000 7f008f172640 1 --2- 172.21.15.12:0/1456838890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d3270 0x7f00880cf370 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.014+0000 7f008f172640 1 --2- 172.21.15.12:0/1456838890 >> v2:172.21.15.55:3300/0 conn(0x7f008815e700 0x7f00880b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.014+0000 7f008f172640 1 -- 172.21.15.12:0/1456838890 >> 172.21.15.12:0/1456838890 conn(0x7f008808bf60 msgr2=0x7f00880d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.014+0000 7f008f172640 1 -- 172.21.15.12:0/1456838890 shutdown_connections 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.014+0000 7f008f172640 1 -- 172.21.15.12:0/1456838890 wait complete. 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.015+0000 7f008f172640 1 Processor -- start 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.015+0000 7f008f172640 1 -- start start 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008f172640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d2360 0x7f008815cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008f172640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00880d3270 0x7f008815aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008f172640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f008815e700 0x7f00880b9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008f172640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f00880b5d80 con 0x7f00880d2360 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008f172640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f00880b4450 con 0x7f00880d3270 2024-04-15T15:21:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008f172640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f00880d1d40 con 0x7f008815e700 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008cf0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d2360 0x7f008815cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008cf0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d2360 0x7f008815cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52614/0 (socket says 172.21.15.12:52614) 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008cf0e640 1 -- 172.21.15.12:0/759122426 learned_addr learned my addr 172.21.15.12:0/759122426 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008cf0e640 1 -- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f008815e700 msgr2=0x7f00880b9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008cf0e640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f008815e700 0x7f00880b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008cf0e640 1 -- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00880d3270 msgr2=0x7f008815aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f0087fff640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00880d3270 0x7f008815aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008cf0e640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00880d3270 0x7f008815aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f008cf0e640 1 -- 172.21.15.12:0/759122426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00880b7180 con 0x7f00880d2360 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.016+0000 7f0087fff640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00880d3270 0x7f008815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.017+0000 7f008cf0e640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d2360 0x7f008815cc40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0080007ef0 tx=0x7f008000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.017+0000 7f00857fa640 1 -- 172.21.15.12:0/759122426 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008001a070 con 0x7f00880d2360 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.017+0000 7f00857fa640 1 -- 172.21.15.12:0/759122426 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0080003b90 con 0x7f00880d2360 2024-04-15T15:21:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.017+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00880b7440 con 0x7f00880d2360 2024-04-15T15:21:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.017+0000 7f00857fa640 1 -- 172.21.15.12:0/759122426 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080004030 con 0x7f00880d2360 2024-04-15T15:21:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.017+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00880b7a00 con 0x7f00880d2360 2024-04-15T15:21:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.018+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00880689e0 con 0x7f00880d2360 2024-04-15T15:21:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.018+0000 7f00857fa640 1 -- 172.21.15.12:0/759122426 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f008001d070 con 0x7f00880d2360 2024-04-15T15:21:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.019+0000 7f00857fa640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f007406c8f0 0x7f007406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.019+0000 7f00857fa640 1 -- 172.21.15.12:0/759122426 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f008008c170 con 0x7f00880d2360 2024-04-15T15:21:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.019+0000 7f0087fff640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f007406c8f0 0x7f007406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.019+0000 7f0087fff640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f007406c8f0 0x7f007406edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f00880c7e60 tx=0x7f007c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.023+0000 7f00857fa640 1 -- 172.21.15.12:0/759122426 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0080010070 con 0x7f00880d2360 2024-04-15T15:21:55.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.207+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f00880ba500 con 0x7f00880d2360 2024-04-15T15:21:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.207+0000 7f00857fa640 1 -- 172.21.15.12:0/759122426 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f0080059ba0 con 0x7f00880d2360 2024-04-15T15:21:55.209 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f007406c8f0 msgr2=0x7f007406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f007406c8f0 0x7f007406edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f00880c7e60 tx=0x7f007c009040 comp rx=0 tx=0).stop 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d2360 msgr2=0x7f008815cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d2360 0x7f008815cc40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0080007ef0 tx=0x7f008000d4a0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 shutdown_connections 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f008815e700 0x7f00880b9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f007406c8f0 0x7f007406edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00880d3270 0x7f008815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 --2- 172.21.15.12:0/759122426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00880d2360 0x7f008815cc40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 >> 172.21.15.12:0/759122426 conn(0x7f008808bf60 msgr2=0x7f00880d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 shutdown_connections 2024-04-15T15:21:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.209+0000 7f008f172640 1 -- 172.21.15.12:0/759122426 wait complete. 2024-04-15T15:21:55.222 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:55.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.325+0000 7f27a0cb1640 1 Processor -- start 2024-04-15T15:21:55.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f27a0cb1640 1 -- start start 2024-04-15T15:21:55.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f27a0cb1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f279c15e700 0x7f279c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f27a0cb1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f279c0d2360 0x7f279c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f27a0cb1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279c0b47a0 con 0x7f279c15eae0 2024-04-15T15:21:55.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f27a0cb1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f279c0b53b0 con 0x7f279c0d2360 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f27a0cb1640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f279c0b4a80 con 0x7f279c15e700 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f2793fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f279c0d2360 0x7f279c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f279ad9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f279c15eae0 0x7f279c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33526/0 (socket says 172.21.15.12:33526) 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.326+0000 7f279ad9d640 1 -- 172.21.15.12:0/4127679327 learned_addr learned my addr 172.21.15.12:0/4127679327 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.327+0000 7f279a59c640 1 --2- 172.21.15.12:0/4127679327 >> v2:172.21.15.55:3300/0 conn(0x7f279c15e700 0x7f279c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.327+0000 7f2793fff640 1 -- 172.21.15.12:0/4127679327 >> v2:172.21.15.55:3300/0 conn(0x7f279c15e700 msgr2=0x7f279c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.327+0000 7f2793fff640 1 --2- 172.21.15.12:0/4127679327 >> v2:172.21.15.55:3300/0 conn(0x7f279c15e700 0x7f279c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.327+0000 7f2793fff640 1 -- 172.21.15.12:0/4127679327 >> v1:172.21.15.12:6789/0 conn(0x7f279c15eae0 legacy=0x7f279c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.327+0000 7f2793fff640 1 -- 172.21.15.12:0/4127679327 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279c0b4d60 con 0x7f279c0d2360 2024-04-15T15:21:55.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f2793fff640 1 --2- 172.21.15.12:0/4127679327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c0d2360 0x7f279c0d2b60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f279c06c6c0 tx=0x7f278402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af630f066393be server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f2799d9b640 1 -- 172.21.15.12:0/4127679327 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784035020 con 0x7f279c0d2360 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f2799d9b640 1 -- 172.21.15.12:0/4127679327 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2784031830 con 0x7f279c0d2360 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f2799d9b640 1 -- 172.21.15.12:0/4127679327 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784031c00 con 0x7f279c0d2360 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f27a0cb1640 1 -- 172.21.15.12:0/4127679327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c0d2360 msgr2=0x7f279c0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/4127679327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c0d2360 0x7f279c0d2b60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f279c06c6c0 tx=0x7f278402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f27a0cb1640 1 -- 172.21.15.12:0/4127679327 shutdown_connections 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/4127679327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c0d2360 0x7f279c0d2b60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/4127679327 >> v2:172.21.15.55:3300/0 conn(0x7f279c15e700 0x7f279c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f27a0cb1640 1 -- 172.21.15.12:0/4127679327 >> 172.21.15.12:0/4127679327 conn(0x7f279c08bf60 msgr2=0x7f279c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f27a0cb1640 1 -- 172.21.15.12:0/4127679327 shutdown_connections 2024-04-15T15:21:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.328+0000 7f27a0cb1640 1 -- 172.21.15.12:0/4127679327 wait complete. 2024-04-15T15:21:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.330+0000 7f27a0cb1640 1 Processor -- start 2024-04-15T15:21:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.330+0000 7f27a0cb1640 1 -- start start 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.330+0000 7f27a0cb1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f279c0d2360 0x7f279c0c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.330+0000 7f27a0cb1640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f279c15e700 0x7f279c1f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.330+0000 7f27a0cb1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c15eae0 0x7f279c15a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f27a0cb1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f279c0b5d80 con 0x7f279c15eae0 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f27a0cb1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f279c0b4450 con 0x7f279c0d2360 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f27a0cb1640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f279c0d1d40 con 0x7f279c15e700 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c15eae0 0x7f279c15a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c15eae0 0x7f279c15a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52634/0 (socket says 172.21.15.12:52634) 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 -- 172.21.15.12:0/1518215043 learned_addr learned my addr 172.21.15.12:0/1518215043 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279a59c640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f279c0d2360 0x7f279c0c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 -- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f279c15e700 msgr2=0x7f279c1f7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f279c15e700 0x7f279c1f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 -- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f279c0d2360 msgr2=0x7f279c0c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f279c0d2360 0x7f279c0c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 -- 172.21.15.12:0/1518215043 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279c15fef0 con 0x7f279c15eae0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f279ad9d640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c15eae0 0x7f279c15a780 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f279400cab0 tx=0x7f279400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f2792ffd640 1 -- 172.21.15.12:0/1518215043 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794017070 con 0x7f279c15eae0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f2792ffd640 1 -- 172.21.15.12:0/1518215043 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f279400ac30 con 0x7f279c15eae0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279c160170 con 0x7f279c15eae0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f2792ffd640 1 -- 172.21.15.12:0/1518215043 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794012400 con 0x7f279c15eae0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.331+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f279c160730 con 0x7f279c15eae0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.332+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f279c0689e0 con 0x7f279c15eae0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.333+0000 7f2792ffd640 1 -- 172.21.15.12:0/1518215043 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f279401a070 con 0x7f279c15eae0 2024-04-15T15:21:55.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.333+0000 7f2792ffd640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f277c06c8f0 0x7f277c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.333+0000 7f2792ffd640 1 -- 172.21.15.12:0/1518215043 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f279408bb80 con 0x7f279c15eae0 2024-04-15T15:21:55.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.333+0000 7f279a59c640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f277c06c8f0 0x7f277c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.334+0000 7f279a59c640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f277c06c8f0 0x7f277c06edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f279c161270 tx=0x7f278c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.338+0000 7f2792ffd640 1 -- 172.21.15.12:0/1518215043 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f279c15fef0 con 0x7f279c15eae0 2024-04-15T15:21:55.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.523+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f279c0ba500 con 0x7f279c15eae0 2024-04-15T15:21:55.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.524+0000 7f2792ffd640 1 -- 172.21.15.12:0/1518215043 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f2794059520 con 0x7f279c15eae0 2024-04-15T15:21:55.525 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:55.526 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:55.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f277c06c8f0 msgr2=0x7f277c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f277c06c8f0 0x7f277c06edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f279c161270 tx=0x7f278c008040 comp rx=0 tx=0).stop 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c15eae0 msgr2=0x7f279c15a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c15eae0 0x7f279c15a780 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f279400cab0 tx=0x7f279400cf70 comp rx=0 tx=0).stop 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 shutdown_connections 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f279c15eae0 0x7f279c15a780 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f279c15e700 0x7f279c1f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f277c06c8f0 0x7f277c06edb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 --2- 172.21.15.12:0/1518215043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f279c0d2360 0x7f279c0c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 >> 172.21.15.12:0/1518215043 conn(0x7f279c08bf60 msgr2=0x7f279c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 shutdown_connections 2024-04-15T15:21:55.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.526+0000 7f27a0cb1640 1 -- 172.21.15.12:0/1518215043 wait complete. 2024-04-15T15:21:55.538 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:55.539 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:55.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.645+0000 7f0baf966640 1 Processor -- start 2024-04-15T15:21:55.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.645+0000 7f0baf966640 1 -- start start 2024-04-15T15:21:55.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0baf966640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0ba815e790 0x7f0ba80d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0baf966640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80d1610 0x7f0ba80d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0baf966640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ba80d2480 con 0x7f0ba80cfd20 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0baf966640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0ba80d2db0 con 0x7f0ba815e790 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0baf966640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba80d2760 con 0x7f0ba80d1610 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0bacf01640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0ba80cfd20 0x7f0ba80d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33536/0 (socket says 172.21.15.12:33536) 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0bacf01640 1 -- 172.21.15.12:0/1684664459 learned_addr learned my addr 172.21.15.12:0/1684664459 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0bad702640 1 --2- 172.21.15.12:0/1684664459 >> v2:172.21.15.55:3300/0 conn(0x7f0ba815e790 0x7f0ba80d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0badf03640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80d1610 0x7f0ba80d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2440148004 0 0) 0x7f0ba80d2480 con 0x7f0ba80cfd20 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.646+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b90003520 con 0x7f0ba80cfd20 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3467534314 0 0) 0x7f0b90003520 con 0x7f0ba80cfd20 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80d1610 msgr2=0x7f0ba80d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80d1610 0x7f0ba80d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 >> v2:172.21.15.55:3300/0 conn(0x7f0ba815e790 msgr2=0x7f0ba80d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 --2- 172.21.15.12:0/1684664459 >> v2:172.21.15.55:3300/0 conn(0x7f0ba815e790 0x7f0ba80d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba80d2a40 con 0x7f0ba80cfd20 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f0b9c001670 con 0x7f0ba80cfd20 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 >> v1:172.21.15.12:6789/0 conn(0x7f0ba80cfd20 legacy=0x7f0ba80d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b900039b0 0x7f0b90003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0b90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80d1610 0x7f0b900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba80d2db0 con 0x7f0b90004800 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba80d2760 con 0x7f0b900039b0 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9000b9c0 con 0x7f0ba80d1610 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0bacf01640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0b90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0bad702640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b900039b0 0x7f0b90003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0badf03640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80d1610 0x7f0b900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0bacf01640 1 -- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80d1610 msgr2=0x7f0b900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0bacf01640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80d1610 0x7f0b900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0bacf01640 1 -- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b900039b0 msgr2=0x7f0b90003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0bacf01640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b900039b0 0x7f0b90003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.647+0000 7f0bacf01640 1 -- 172.21.15.12:0/1684664459 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b90003520 con 0x7f0b90004800 2024-04-15T15:21:55.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.648+0000 7f0bacf01640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0b90004c80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00c240 tx=0x7f0b90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc85ecea86f65072 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.648+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c007d30 con 0x7f0b90004800 2024-04-15T15:21:55.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.648+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9c030070 con 0x7f0b90004800 2024-04-15T15:21:55.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.648+0000 7f0ba7fff640 1 -- 172.21.15.12:0/1684664459 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c02cd60 con 0x7f0b90004800 2024-04-15T15:21:55.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.649+0000 7f0baf966640 1 -- 172.21.15.12:0/1684664459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 msgr2=0x7f0b90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.649+0000 7f0baf966640 1 --2- 172.21.15.12:0/1684664459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0b90004c80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00c240 tx=0x7f0b90005830 comp rx=0 tx=0).stop 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.649+0000 7f0badf03640 1 -- 172.21.15.12:0/1684664459 reap_dead start 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.649+0000 7f0baf966640 1 -- 172.21.15.12:0/1684664459 shutdown_connections 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.649+0000 7f0baf966640 1 -- 172.21.15.12:0/1684664459 >> 172.21.15.12:0/1684664459 conn(0x7f0ba808bf60 msgr2=0x7f0ba80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.649+0000 7f0baf966640 1 -- 172.21.15.12:0/1684664459 shutdown_connections 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.649+0000 7f0baf966640 1 -- 172.21.15.12:0/1684664459 wait complete. 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.650+0000 7f0baf966640 1 Processor -- start 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.650+0000 7f0baf966640 1 -- start start 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0baf966640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0ba8152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0baf966640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ba80c91e0 0x7f0ba80b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0baf966640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80c9640 0x7f0ba80b8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0baf966640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba815ecf0 con 0x7f0b90004800 2024-04-15T15:21:55.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0baf966640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba815eb70 con 0x7f0ba80c91e0 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0baf966640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba80d0160 con 0x7f0ba80c9640 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0ba8152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0ba8152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52646/0 (socket says 172.21.15.12:52646) 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 -- 172.21.15.12:0/3387893882 learned_addr learned my addr 172.21.15.12:0/3387893882 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0badf03640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80c9640 0x7f0ba80b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bacf01640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ba80c91e0 0x7f0ba80b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 -- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80c9640 msgr2=0x7f0ba80b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80c9640 0x7f0ba80b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 -- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ba80c91e0 msgr2=0x7f0ba80b81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ba80c91e0 0x7f0ba80b81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 -- 172.21.15.12:0/3387893882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba81530d0 con 0x7f0b90004800 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0badf03640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80c9640 0x7f0ba80b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:55.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0bad702640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0ba8152b90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0011960 tx=0x7f0ba0011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0ba5ffb640 1 -- 172.21.15.12:0/3387893882 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba000faf0 con 0x7f0b90004800 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0ba5ffb640 1 -- 172.21.15.12:0/3387893882 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba000fc90 con 0x7f0b90004800 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba80c9c10 con 0x7f0b90004800 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.651+0000 7f0ba5ffb640 1 -- 172.21.15.12:0/3387893882 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0014440 con 0x7f0b90004800 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.652+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba80ca1d0 con 0x7f0b90004800 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.653+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba80689e0 con 0x7f0b90004800 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.653+0000 7f0ba5ffb640 1 -- 172.21.15.12:0/3387893882 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ba0012040 con 0x7f0b90004800 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.653+0000 7f0ba5ffb640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b9406c8f0 0x7f0b9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.654+0000 7f0ba5ffb640 1 -- 172.21.15.12:0/3387893882 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f0ba008c610 con 0x7f0b90004800 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.654+0000 7f0bacf01640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b9406c8f0 0x7f0b9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.654+0000 7f0bacf01640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b9406c8f0 0x7f0b9406edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c034000 tx=0x7f0b9c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.658+0000 7f0ba5ffb640 1 -- 172.21.15.12:0/3387893882 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ba0019030 con 0x7f0b90004800 2024-04-15T15:21:55.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.844+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0ba80d1ce0 con 0x7f0b90004800 2024-04-15T15:21:55.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.844+0000 7f0ba5ffb640 1 -- 172.21.15.12:0/3387893882 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f0ba005a040 con 0x7f0b90004800 2024-04-15T15:21:55.846 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:55.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b9406c8f0 msgr2=0x7f0b9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b9406c8f0 0x7f0b9406edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c034000 tx=0x7f0b9c007000 comp rx=0 tx=0).stop 2024-04-15T15:21:55.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 msgr2=0x7f0ba8152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0ba8152b90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0011960 tx=0x7f0ba0011e20 comp rx=0 tx=0).stop 2024-04-15T15:21:55.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 shutdown_connections 2024-04-15T15:21:55.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ba80c9640 0x7f0ba80b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b9406c8f0 0x7f0b9406edb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ba80c91e0 0x7f0ba80b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 --2- 172.21.15.12:0/3387893882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b90004800 0x7f0ba8152b90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 >> 172.21.15.12:0/3387893882 conn(0x7f0ba808bf60 msgr2=0x7f0ba8074dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:55.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 shutdown_connections 2024-04-15T15:21:55.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.846+0000 7f0baf966640 1 -- 172.21.15.12:0/3387893882 wait complete. 2024-04-15T15:21:55.859 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.967+0000 7fc0cdfa3640 1 Processor -- start 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.967+0000 7fc0cdfa3640 1 -- start start 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.967+0000 7fc0cdfa3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc0c80d5150 0x7fc0c80d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.967+0000 7fc0cdfa3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c815eb70 con 0x7fc0c80d15e0 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.967+0000 7fc0cdfa3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c80d3c80 con 0x7fc0c815e790 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.967+0000 7fc0cdfa3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc0c80d3790 con 0x7fc0c80d5150 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c6ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc0c80d15e0 0x7fc0c80d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33546/0 (socket says 172.21.15.12:33546) 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c6ffd640 1 -- 172.21.15.12:0/2082897844 learned_addr learned my addr 172.21.15.12:0/2082897844 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2235234430 0 0) 0x7fc0c815eb70 con 0x7fc0c80d15e0 2024-04-15T15:21:55.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0b4003520 con 0x7fc0c80d15e0 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c7fff640 1 --2- 172.21.15.12:0/2082897844 >> v2:172.21.15.55:3300/0 conn(0x7fc0c80d5150 0x7fc0c80d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1187551865 0 0) 0x7fc0b4003520 con 0x7fc0c80d15e0 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 >> v2:172.21.15.55:3300/0 conn(0x7fc0c80d5150 msgr2=0x7fc0c80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 --2- 172.21.15.12:0/2082897844 >> v2:172.21.15.55:3300/0 conn(0x7fc0c80d5150 0x7fc0c80d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 >> v1:172.21.15.55:6789/0 conn(0x7fc0c815e790 legacy=0x7fc0c80b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c80d3f60 con 0x7fc0c80d15e0 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fc0b0001670 con 0x7fc0c80d15e0 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 >> v1:172.21.15.12:6789/0 conn(0x7fc0c80d15e0 legacy=0x7fc0c80d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b40039b0 0x7fc0b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0b40053c0 0x7fc0b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c80d3790 con 0x7fc0b4004800 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b400c250 con 0x7fc0b40053c0 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc0b400c450 con 0x7fc0b40039b0 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c6ffd640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c7fff640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0b40053c0 0x7fc0b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c77fe640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b40039b0 0x7fc0b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c6ffd640 1 -- 172.21.15.12:0/2082897844 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b40039b0 msgr2=0x7fc0b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c6ffd640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b40039b0 0x7fc0b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c6ffd640 1 -- 172.21.15.12:0/2082897844 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0b40053c0 msgr2=0x7fc0b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c6ffd640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0b40053c0 0x7fc0b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c6ffd640 1 -- 172.21.15.12:0/2082897844 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b4003520 con 0x7fc0b4004800 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.968+0000 7fc0c7fff640 1 -- 172.21.15.12:0/2082897844 reap_dead start 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.969+0000 7fc0c6ffd640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000c240 tx=0x7fc0b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46b95fb94dcc9a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.969+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0007cf0 con 0x7fc0b4004800 2024-04-15T15:21:55.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.969+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b0030070 con 0x7fc0b4004800 2024-04-15T15:21:55.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.969+0000 7fc0c67fc640 1 -- 172.21.15.12:0/2082897844 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0005d50 con 0x7fc0b4004800 2024-04-15T15:21:55.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.970+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/2082897844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b4004800 msgr2=0x7fc0b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.970+0000 7fc0cdfa3640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000c240 tx=0x7fc0b4005830 comp rx=0 tx=0).stop 2024-04-15T15:21:55.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.970+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/2082897844 shutdown_connections 2024-04-15T15:21:55.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.970+0000 7fc0cdfa3640 1 --2- 172.21.15.12:0/2082897844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.970+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/2082897844 >> 172.21.15.12:0/2082897844 conn(0x7fc0c8067b40 msgr2=0x7fc0c8067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:55.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.970+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/2082897844 shutdown_connections 2024-04-15T15:21:55.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.970+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/2082897844 wait complete. 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.971+0000 7fc0cdfa3640 1 Processor -- start 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.971+0000 7fc0cdfa3640 1 -- start start 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0cdfa3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b4004800 0x7fc0c80caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0cdfa3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c81e89e0 0x7fc0c815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0cdfa3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0c815cf10 0x7fc0c81f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0cdfa3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c80d1a90 con 0x7fc0c815cf10 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0cdfa3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c80d45b0 con 0x7fc0c81e89e0 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0cdfa3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c80d1c10 con 0x7fc0b4004800 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0c815cf10 0x7fc0c81f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0c815cf10 0x7fc0c81f74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52670/0 (socket says 172.21.15.12:52670) 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c7fff640 1 -- 172.21.15.12:0/484736034 learned_addr learned my addr 172.21.15.12:0/484736034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c6ffd640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c81e89e0 0x7fc0c815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c7fff640 1 -- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b4004800 msgr2=0x7fc0c80caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c77fe640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b4004800 0x7fc0c80caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c7fff640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b4004800 0x7fc0c80caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c7fff640 1 -- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c81e89e0 msgr2=0x7fc0c815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c7fff640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c81e89e0 0x7fc0c815c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.972+0000 7fc0c7fff640 1 -- 172.21.15.12:0/484736034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c8147260 con 0x7fc0c815cf10 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.973+0000 7fc0c7fff640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0c815cf10 0x7fc0c81f74f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80d2440 tx=0x7fc0c0009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.973+0000 7fc0bffff640 1 -- 172.21.15.12:0/484736034 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0007c70 con 0x7fc0c815cf10 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.973+0000 7fc0bffff640 1 -- 172.21.15.12:0/484736034 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0c0019070 con 0x7fc0c815cf10 2024-04-15T15:21:55.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.973+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c8147540 con 0x7fc0c815cf10 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.973+0000 7fc0bffff640 1 -- 172.21.15.12:0/484736034 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0014650 con 0x7fc0c815cf10 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.973+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c8147b00 con 0x7fc0c815cf10 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.974+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c80689e0 con 0x7fc0c815cf10 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.974+0000 7fc0bffff640 1 -- 172.21.15.12:0/484736034 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0c00147f0 con 0x7fc0c815cf10 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.975+0000 7fc0bffff640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0a806c8f0 0x7fc0a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.975+0000 7fc0bffff640 1 -- 172.21.15.12:0/484736034 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fc0c008c4f0 con 0x7fc0c815cf10 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.975+0000 7fc0c77fe640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0a806c8f0 0x7fc0a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.975+0000 7fc0c77fe640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0a806c8f0 0x7fc0a806edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8007fd0 tx=0x7fc0b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:55.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:55.979+0000 7fc0bffff640 1 -- 172.21.15.12:0/484736034 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0c005a4e0 con 0x7fc0c815cf10 2024-04-15T15:21:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.162+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc0c80cafc0 con 0x7fc0c815cf10 2024-04-15T15:21:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.162+0000 7fc0bffff640 1 -- 172.21.15.12:0/484736034 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fc0c0059de0 con 0x7fc0c815cf10 2024-04-15T15:21:56.164 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0a806c8f0 msgr2=0x7fc0a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0a806c8f0 0x7fc0a806edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8007fd0 tx=0x7fc0b8009040 comp rx=0 tx=0).stop 2024-04-15T15:21:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0c815cf10 msgr2=0x7fc0c81f74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0c815cf10 0x7fc0c81f74f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80d2440 tx=0x7fc0c0009cd0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 shutdown_connections 2024-04-15T15:21:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0c815cf10 0x7fc0c81f74f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0a806c8f0 0x7fc0a806edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c81e89e0 0x7fc0c815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 --2- 172.21.15.12:0/484736034 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0b4004800 0x7fc0c80caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 >> 172.21.15.12:0/484736034 conn(0x7fc0c8067b40 msgr2=0x7fc0c80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 shutdown_connections 2024-04-15T15:21:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.164+0000 7fc0cdfa3640 1 -- 172.21.15.12:0/484736034 wait complete. 2024-04-15T15:21:56.177 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:56.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.283+0000 7f25005b5640 1 Processor -- start 2024-04-15T15:21:56.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.283+0000 7f25005b5640 1 -- start start 2024-04-15T15:21:56.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f25005b5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f24f80d15e0 0x7f24f80d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f25005b5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24f80d4a40 0x7f24f80d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f25005b5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f815eb70 con 0x7f24f815e790 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f25005b5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f24f80d3790 con 0x7f24f80d4a40 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f25005b5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f24f80d45b0 con 0x7f24f80d15e0 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f24feb52640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f24f80d4a40 0x7f24f80d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f24fe351640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f24f815e790 0x7f24f80b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33556/0 (socket says 172.21.15.12:33556) 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f24fe351640 1 -- 172.21.15.12:0/1179616392 learned_addr learned my addr 172.21.15.12:0/1179616392 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f24fdb50640 1 --2- 172.21.15.12:0/1179616392 >> v2:172.21.15.55:3300/0 conn(0x7f24f80d15e0 0x7f24f80d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f24feb52640 1 -- 172.21.15.12:0/1179616392 >> v2:172.21.15.55:3300/0 conn(0x7f24f80d15e0 msgr2=0x7f24f80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f24feb52640 1 --2- 172.21.15.12:0/1179616392 >> v2:172.21.15.55:3300/0 conn(0x7f24f80d15e0 0x7f24f80d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.284+0000 7f24feb52640 1 -- 172.21.15.12:0/1179616392 >> v1:172.21.15.12:6789/0 conn(0x7f24f815e790 legacy=0x7f24f80b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.285+0000 7f24fd34f640 1 -- 172.21.15.12:0/1179616392 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916308997 0 0) 0x7f24f815eb70 con 0x7f24f815e790 2024-04-15T15:21:56.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.285+0000 7f24feb52640 1 -- 172.21.15.12:0/1179616392 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f80d3c80 con 0x7f24f80d4a40 2024-04-15T15:21:56.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.285+0000 7f24fdb50640 1 --2- 172.21.15.12:0/1179616392 >> v2:172.21.15.55:3300/0 conn(0x7f24f80d15e0 0x7f24f80d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:56.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.285+0000 7f24feb52640 1 --2- 172.21.15.12:0/1179616392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d4a40 0x7f24f80d4eb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f24f80d5690 tx=0x7f24e802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bd874ba7067c43 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.285+0000 7f24fd34f640 1 -- 172.21.15.12:0/1179616392 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8039070 con 0x7f24f80d4a40 2024-04-15T15:21:56.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.285+0000 7f24fd34f640 1 -- 172.21.15.12:0/1179616392 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e802cb60 con 0x7f24f80d4a40 2024-04-15T15:21:56.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.285+0000 7f24fd34f640 1 -- 172.21.15.12:0/1179616392 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8034440 con 0x7f24f80d4a40 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.286+0000 7f25005b5640 1 -- 172.21.15.12:0/1179616392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d4a40 msgr2=0x7f24f80d4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.286+0000 7f25005b5640 1 --2- 172.21.15.12:0/1179616392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d4a40 0x7f24f80d4eb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f24f80d5690 tx=0x7f24e802bdb0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.286+0000 7f25005b5640 1 -- 172.21.15.12:0/1179616392 shutdown_connections 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.286+0000 7f25005b5640 1 --2- 172.21.15.12:0/1179616392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d4a40 0x7f24f80d4eb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.286+0000 7f25005b5640 1 --2- 172.21.15.12:0/1179616392 >> v2:172.21.15.55:3300/0 conn(0x7f24f80d15e0 0x7f24f80d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.286+0000 7f25005b5640 1 -- 172.21.15.12:0/1179616392 >> 172.21.15.12:0/1179616392 conn(0x7f24f8067b40 msgr2=0x7f24f8067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.286+0000 7f25005b5640 1 -- 172.21.15.12:0/1179616392 shutdown_connections 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.286+0000 7f25005b5640 1 -- 172.21.15.12:0/1179616392 wait complete. 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.287+0000 7f25005b5640 1 Processor -- start 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.287+0000 7f25005b5640 1 -- start start 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f25005b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d15e0 0x7f24f81f9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f25005b5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f24f80d4a40 0x7f24f80b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f25005b5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24f815e790 0x7f24f80cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f25005b5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f24f80d4240 con 0x7f24f80d15e0 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f25005b5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f24f80d53f0 con 0x7f24f815e790 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f25005b5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f24f80b7db0 con 0x7f24f80d4a40 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f24fe351640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d15e0 0x7f24f81f9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f24fe351640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d15e0 0x7f24f81f9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52674/0 (socket says 172.21.15.12:52674) 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f24fe351640 1 -- 172.21.15.12:0/2406971464 learned_addr learned my addr 172.21.15.12:0/2406971464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f24fdb50640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f24f80d4a40 0x7f24f80b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.288+0000 7f24feb52640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24f815e790 0x7f24f80cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24fe351640 1 -- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f24f80d4a40 msgr2=0x7f24f80b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24fe351640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f24f80d4a40 0x7f24f80b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24fe351640 1 -- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24f815e790 msgr2=0x7f24f80cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24fe351640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24f815e790 0x7f24f80cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24fe351640 1 -- 172.21.15.12:0/2406971464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f81f2aa0 con 0x7f24f80d15e0 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24fdb50640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f24f80d4a40 0x7f24f80b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24feb52640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24f815e790 0x7f24f80cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24fe351640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d15e0 0x7f24f81f9ab0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f24f000c4d0 tx=0x7f24f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24f6ffd640 1 -- 172.21.15.12:0/2406971464 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f0017070 con 0x7f24f80d15e0 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24f6ffd640 1 -- 172.21.15.12:0/2406971464 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24f0005d00 con 0x7f24f80d15e0 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f81f8af0 con 0x7f24f80d15e0 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f24f6ffd640 1 -- 172.21.15.12:0/2406971464 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f00072b0 con 0x7f24f80d15e0 2024-04-15T15:21:56.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.289+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f81f90b0 con 0x7f24f80d15e0 2024-04-15T15:21:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.290+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f80689e0 con 0x7f24f80d15e0 2024-04-15T15:21:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.291+0000 7f24f6ffd640 1 -- 172.21.15.12:0/2406971464 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f24f001a070 con 0x7f24f80d15e0 2024-04-15T15:21:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.291+0000 7f24f6ffd640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f24e406c8f0 0x7f24e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.291+0000 7f24f6ffd640 1 -- 172.21.15.12:0/2406971464 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f24f008b330 con 0x7f24f80d15e0 2024-04-15T15:21:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.291+0000 7f24fdb50640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f24e406c8f0 0x7f24e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.292+0000 7f24fdb50640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f24e406c8f0 0x7f24e406edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f24ec006fd0 tx=0x7f24ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.296+0000 7f24f6ffd640 1 -- 172.21.15.12:0/2406971464 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f24f81f2aa0 con 0x7f24f80d15e0 2024-04-15T15:21:56.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.480+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f24f8161200 con 0x7f24f80d15e0 2024-04-15T15:21:56.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.481+0000 7f24f6ffd640 1 -- 172.21.15.12:0/2406971464 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f24f0059320 con 0x7f24f80d15e0 2024-04-15T15:21:56.482 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:56.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f24e406c8f0 msgr2=0x7f24e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f24e406c8f0 0x7f24e406edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f24ec006fd0 tx=0x7f24ec008040 comp rx=0 tx=0).stop 2024-04-15T15:21:56.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d15e0 msgr2=0x7f24f81f9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d15e0 0x7f24f81f9ab0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f24f000c4d0 tx=0x7f24f000c990 comp rx=0 tx=0).stop 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 shutdown_connections 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f24e406c8f0 0x7f24e406edb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24f815e790 0x7f24f80cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f24f80d4a40 0x7f24f80b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 --2- 172.21.15.12:0/2406971464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24f80d15e0 0x7f24f81f9ab0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 >> 172.21.15.12:0/2406971464 conn(0x7f24f8067b40 msgr2=0x7f24f80b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 shutdown_connections 2024-04-15T15:21:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.483+0000 7f25005b5640 1 -- 172.21.15.12:0/2406971464 wait complete. 2024-04-15T15:21:56.495 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:56.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.601+0000 7f0c61cc8640 1 Processor -- start 2024-04-15T15:21:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.601+0000 7f0c61cc8640 1 -- start start 2024-04-15T15:21:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c61cc8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0c5c0b3160 0x7f0c5c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c61cc8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0c5c0ae280 0x7f0c5c0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c61cc8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5c15cb50 con 0x7f0c5c0d0690 2024-04-15T15:21:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c61cc8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5c15bfd0 con 0x7f0c5c0ae280 2024-04-15T15:21:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c61cc8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5c15c2b0 con 0x7f0c5c0b3160 2024-04-15T15:21:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c5bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0c5c0ae280 0x7f0c5c0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c5bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0c5c0ae280 0x7f0c5c0ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52686/0 (socket says 172.21.15.12:52686) 2024-04-15T15:21:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c5bfff640 1 -- 172.21.15.12:0/3954095240 learned_addr learned my addr 172.21.15.12:0/3954095240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:56.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c53fff640 1 --2- 172.21.15.12:0/3954095240 >> v2:172.21.15.55:3300/0 conn(0x7f0c5c0b3160 0x7f0c5c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c5bfff640 1 -- 172.21.15.12:0/3954095240 >> v2:172.21.15.55:3300/0 conn(0x7f0c5c0b3160 msgr2=0x7f0c5c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:56.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c5bfff640 1 --2- 172.21.15.12:0/3954095240 >> v2:172.21.15.55:3300/0 conn(0x7f0c5c0b3160 0x7f0c5c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c5bfff640 1 -- 172.21.15.12:0/3954095240 >> v1:172.21.15.55:6789/0 conn(0x7f0c5c0d0690 legacy=0x7f0c5c0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.602+0000 7f0c5bfff640 1 -- 172.21.15.12:0/3954095240 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c15c590 con 0x7f0c5c0ae280 2024-04-15T15:21:56.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.603+0000 7f0c5bfff640 1 --2- 172.21.15.12:0/3954095240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0ae710 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06fec0 tx=0x7f0c5402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68eead7077c2f87 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.603+0000 7f0c5affd640 1 -- 172.21.15.12:0/3954095240 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5402cb20 con 0x7f0c5c0ae280 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.603+0000 7f0c5affd640 1 -- 172.21.15.12:0/3954095240 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5402ccc0 con 0x7f0c5c0ae280 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.603+0000 7f0c5affd640 1 -- 172.21.15.12:0/3954095240 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c54034610 con 0x7f0c5c0ae280 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.604+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3954095240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 msgr2=0x7f0c5c0ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.604+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3954095240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0ae710 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06fec0 tx=0x7f0c5402bf80 comp rx=0 tx=0).stop 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.604+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3954095240 shutdown_connections 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.604+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3954095240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0ae710 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.604+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3954095240 >> v2:172.21.15.55:3300/0 conn(0x7f0c5c0b3160 0x7f0c5c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.604+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3954095240 >> 172.21.15.12:0/3954095240 conn(0x7f0c5c06be00 msgr2=0x7f0c5c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.604+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3954095240 shutdown_connections 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.604+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3954095240 wait complete. 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.605+0000 7f0c61cc8640 1 Processor -- start 2024-04-15T15:21:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.605+0000 7f0c61cc8640 1 -- start start 2024-04-15T15:21:56.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c61cc8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c61cc8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0b3160 0x7f0c5c167900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c61cc8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c5c0d0690 0x7f0c5c14ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c61cc8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c15bbd0 con 0x7f0c5c0ae280 2024-04-15T15:21:56.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c61cc8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c0b4fb0 con 0x7f0c5c0b3160 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c61cc8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5c15bd50 con 0x7f0c5c0d0690 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52698/0 (socket says 172.21.15.12:52698) 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 -- 172.21.15.12:0/3169615867 learned_addr learned my addr 172.21.15.12:0/3169615867 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c53fff640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0b3160 0x7f0c5c167900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5bfff640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c5c0d0690 0x7f0c5c14ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 -- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c5c0d0690 msgr2=0x7f0c5c14ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c5c0d0690 0x7f0c5c14ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 -- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0b3160 msgr2=0x7f0c5c167900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0b3160 0x7f0c5c167900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 -- 172.21.15.12:0/3169615867 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c168150 con 0x7f0c5c0ae280 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5bfff640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c5c0d0690 0x7f0c5c14ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c53fff640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0b3160 0x7f0c5c167900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c5b7fe640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0b12f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00c3c0 tx=0x7f0c4c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c58ff9640 1 -- 172.21.15.12:0/3169615867 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c0052d0 con 0x7f0c5c0ae280 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c58ff9640 1 -- 172.21.15.12:0/3169615867 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4c005490 con 0x7f0c5c0ae280 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.606+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5c168430 con 0x7f0c5c0ae280 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.607+0000 7f0c58ff9640 1 -- 172.21.15.12:0/3169615867 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c007260 con 0x7f0c5c0ae280 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.607+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5c1689f0 con 0x7f0c5c0ae280 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.607+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c5c0689e0 con 0x7f0c5c0ae280 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.608+0000 7f0c58ff9640 1 -- 172.21.15.12:0/3169615867 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c4c007400 con 0x7f0c5c0ae280 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.608+0000 7f0c58ff9640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c3c06c8f0 0x7f0c3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.609+0000 7f0c58ff9640 1 -- 172.21.15.12:0/3169615867 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f0c4c08b3c0 con 0x7f0c5c0ae280 2024-04-15T15:21:56.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.609+0000 7f0c53fff640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c3c06c8f0 0x7f0c3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.609+0000 7f0c53fff640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c3c06c8f0 0x7f0c3c06edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06dca0 tx=0x7f0c44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.613+0000 7f0c58ff9640 1 -- 172.21.15.12:0/3169615867 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c4c059440 con 0x7f0c5c0ae280 2024-04-15T15:21:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.796+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0c5c1670f0 con 0x7f0c5c0ae280 2024-04-15T15:21:56.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.797+0000 7f0c58ff9640 1 -- 172.21.15.12:0/3169615867 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f0c4c058d40 con 0x7f0c5c0ae280 2024-04-15T15:21:56.798 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c3c06c8f0 msgr2=0x7f0c3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c3c06c8f0 0x7f0c3c06edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06dca0 tx=0x7f0c44008040 comp rx=0 tx=0).stop 2024-04-15T15:21:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 msgr2=0x7f0c5c0b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0b12f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00c3c0 tx=0x7f0c4c00c880 comp rx=0 tx=0).stop 2024-04-15T15:21:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 shutdown_connections 2024-04-15T15:21:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c5c0d0690 0x7f0c5c14ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c3c06c8f0 0x7f0c3c06edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0b3160 0x7f0c5c167900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 --2- 172.21.15.12:0/3169615867 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5c0ae280 0x7f0c5c0b12f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 >> 172.21.15.12:0/3169615867 conn(0x7f0c5c06be00 msgr2=0x7f0c5c0b3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 shutdown_connections 2024-04-15T15:21:56.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.799+0000 7f0c61cc8640 1 -- 172.21.15.12:0/3169615867 wait complete. 2024-04-15T15:21:56.812 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:56.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.916+0000 7f7ca55a8640 1 Processor -- start 2024-04-15T15:21:56.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.916+0000 7f7ca55a8640 1 -- start start 2024-04-15T15:21:56.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.916+0000 7f7ca55a8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7ca015c8d0 0x7f7ca00cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.916+0000 7f7ca55a8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ca009b650 0x7f7ca009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.916+0000 7f7ca55a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ca00d0350 con 0x7f7ca009af10 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.916+0000 7f7ca55a8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7ca00b5b00 con 0x7f7ca009b650 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.916+0000 7f7ca55a8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7ca009b340 con 0x7f7ca015c8d0 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.917+0000 7f7c97fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ca009b650 0x7f7ca009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.917+0000 7f7c97fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ca009b650 0x7f7ca009a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52702/0 (socket says 172.21.15.12:52702) 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.917+0000 7f7c97fff640 1 -- 172.21.15.12:0/1244588294 learned_addr learned my addr 172.21.15.12:0/1244588294 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.917+0000 7f7c9effd640 1 --2- 172.21.15.12:0/1244588294 >> v2:172.21.15.55:3300/0 conn(0x7f7ca015c8d0 0x7f7ca00cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.917+0000 7f7c97fff640 1 -- 172.21.15.12:0/1244588294 >> v2:172.21.15.55:3300/0 conn(0x7f7ca015c8d0 msgr2=0x7f7ca00cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.917+0000 7f7c97fff640 1 --2- 172.21.15.12:0/1244588294 >> v2:172.21.15.55:3300/0 conn(0x7f7ca015c8d0 0x7f7ca00cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.917+0000 7f7c97fff640 1 -- 172.21.15.12:0/1244588294 >> v1:172.21.15.55:6789/0 conn(0x7f7ca009af10 legacy=0x7f7ca00d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:56.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.917+0000 7f7c97fff640 1 -- 172.21.15.12:0/1244588294 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca00ce100 con 0x7f7ca009b650 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7c97fff640 1 --2- 172.21.15.12:0/1244588294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca009b650 0x7f7ca009a9d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7ca009b620 tx=0x7f7c8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ffcdd6c246dd5e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7c9e7fc640 1 -- 172.21.15.12:0/1244588294 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8802e070 con 0x7f7ca009b650 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7c9e7fc640 1 -- 172.21.15.12:0/1244588294 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c88037070 con 0x7f7ca009b650 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7c9e7fc640 1 -- 172.21.15.12:0/1244588294 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c88031a00 con 0x7f7ca009b650 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7ca55a8640 1 -- 172.21.15.12:0/1244588294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca009b650 msgr2=0x7f7ca009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/1244588294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca009b650 0x7f7ca009a9d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7ca009b620 tx=0x7f7c8802b880 comp rx=0 tx=0).stop 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7ca55a8640 1 -- 172.21.15.12:0/1244588294 shutdown_connections 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/1244588294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca009b650 0x7f7ca009a9d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/1244588294 >> v2:172.21.15.55:3300/0 conn(0x7f7ca015c8d0 0x7f7ca00cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7ca55a8640 1 -- 172.21.15.12:0/1244588294 >> 172.21.15.12:0/1244588294 conn(0x7f7ca00b3fb0 msgr2=0x7f7ca00b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:56.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.918+0000 7f7ca55a8640 1 -- 172.21.15.12:0/1244588294 shutdown_connections 2024-04-15T15:21:56.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.919+0000 7f7ca55a8640 1 -- 172.21.15.12:0/1244588294 wait complete. 2024-04-15T15:21:56.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.920+0000 7f7ca55a8640 1 Processor -- start 2024-04-15T15:21:56.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.920+0000 7f7ca55a8640 1 -- start start 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.920+0000 7f7ca55a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ca009af10 0x7f7ca00c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.920+0000 7f7ca55a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ca009b650 0x7f7ca00c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.920+0000 7f7ca55a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca01535b0 0x7f7ca0071080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.920+0000 7f7ca55a8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca00b6fa0 con 0x7f7ca01535b0 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.920+0000 7f7ca55a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca00b5790 con 0x7f7ca009b650 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.920+0000 7f7ca55a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca00b7120 con 0x7f7ca009af10 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca01535b0 0x7f7ca0071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca01535b0 0x7f7ca0071080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52706/0 (socket says 172.21.15.12:52706) 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 -- 172.21.15.12:0/2860412834 learned_addr learned my addr 172.21.15.12:0/2860412834 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 -- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ca009af10 msgr2=0x7f7ca00c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ca009af10 0x7f7ca00c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c97fff640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ca009b650 0x7f7ca00c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 -- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ca009b650 msgr2=0x7f7ca00c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ca009b650 0x7f7ca00c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 -- 172.21.15.12:0/2860412834 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca00c9ad0 con 0x7f7ca01535b0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c9f7fe640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca01535b0 0x7f7ca0071080 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b970 tx=0x7f7c9800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c977fe640 1 -- 172.21.15.12:0/2860412834 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98019070 con 0x7f7ca01535b0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c977fe640 1 -- 172.21.15.12:0/2860412834 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c9800cb40 con 0x7f7ca01535b0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca00c9d50 con 0x7f7ca01535b0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7c977fe640 1 -- 172.21.15.12:0/2860412834 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98014440 con 0x7f7ca01535b0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.921+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca00ca310 con 0x7f7ca01535b0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.922+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca00cd1f0 con 0x7f7ca01535b0 2024-04-15T15:21:56.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.923+0000 7f7c977fe640 1 -- 172.21.15.12:0/2860412834 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7c9801c070 con 0x7f7ca01535b0 2024-04-15T15:21:56.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.923+0000 7f7c977fe640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c8006c8f0 0x7f7c8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:56.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.923+0000 7f7c977fe640 1 -- 172.21.15.12:0/2860412834 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f7c9808bc00 con 0x7f7ca01535b0 2024-04-15T15:21:56.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.924+0000 7f7c9effd640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c8006c8f0 0x7f7c8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:56.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.924+0000 7f7c9effd640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c8006c8f0 0x7f7c8006edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c900098f0 tx=0x7f7c90008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:56.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:56.928+0000 7f7c977fe640 1 -- 172.21.15.12:0/2860412834 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ca00cd1f0 con 0x7f7ca01535b0 2024-04-15T15:21:57.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.112+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ca00b3d20 con 0x7f7ca01535b0 2024-04-15T15:21:57.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.113+0000 7f7c977fe640 1 -- 172.21.15.12:0/2860412834 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f7ca00b3d20 con 0x7f7ca01535b0 2024-04-15T15:21:57.115 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:57.115 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:57.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c8006c8f0 msgr2=0x7f7c8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c8006c8f0 0x7f7c8006edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c900098f0 tx=0x7f7c90008040 comp rx=0 tx=0).stop 2024-04-15T15:21:57.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca01535b0 msgr2=0x7f7ca0071080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca01535b0 0x7f7ca0071080 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b970 tx=0x7f7c9800be30 comp rx=0 tx=0).stop 2024-04-15T15:21:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 shutdown_connections 2024-04-15T15:21:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ca01535b0 0x7f7ca0071080 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c8006c8f0 0x7f7c8006edb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ca009b650 0x7f7ca00c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 --2- 172.21.15.12:0/2860412834 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ca009af10 0x7f7ca00c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 >> 172.21.15.12:0/2860412834 conn(0x7f7ca00b3fb0 msgr2=0x7f7ca00b3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 shutdown_connections 2024-04-15T15:21:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.115+0000 7f7ca55a8640 1 -- 172.21.15.12:0/2860412834 wait complete. 2024-04-15T15:21:57.128 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:57.129 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.235+0000 7f9595ba8640 1 Processor -- start 2024-04-15T15:21:57.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.235+0000 7f9595ba8640 1 -- start start 2024-04-15T15:21:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f9595ba8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f959015e790 0x7f95900d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f9595ba8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95900d1610 0x7f95900d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f9595ba8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95900d2480 con 0x7f95900cfd20 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f9595ba8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f95900d2db0 con 0x7f959015e790 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f9595ba8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f95900d2760 con 0x7f95900d1610 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f958f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f959015e790 0x7f95900d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f958f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f959015e790 0x7f95900d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52710/0 (socket says 172.21.15.12:52710) 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f958f7fe640 1 -- 172.21.15.12:0/735067743 learned_addr learned my addr 172.21.15.12:0/735067743 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.236+0000 7f958ffff640 1 --2- 172.21.15.12:0/735067743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95900d1610 0x7f95900d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.237+0000 7f958f7fe640 1 -- 172.21.15.12:0/735067743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95900d1610 msgr2=0x7f95900d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.237+0000 7f958f7fe640 1 --2- 172.21.15.12:0/735067743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95900d1610 0x7f95900d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.237+0000 7f958f7fe640 1 -- 172.21.15.12:0/735067743 >> v1:172.21.15.55:6789/0 conn(0x7f95900cfd20 legacy=0x7f95900d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.237+0000 7f958f7fe640 1 -- 172.21.15.12:0/735067743 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95900d2a40 con 0x7f959015e790 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.237+0000 7f958ffff640 1 --2- 172.21.15.12:0/735067743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95900d1610 0x7f95900d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.237+0000 7f958f7fe640 1 --2- 172.21.15.12:0/735067743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f959015e790 0x7f95900d3240 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f95900d2f30 tx=0x7f958002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6d561276a9b8fa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f958e7fc640 1 -- 172.21.15.12:0/735067743 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580034070 con 0x7f959015e790 2024-04-15T15:21:57.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f958e7fc640 1 -- 172.21.15.12:0/735067743 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9580030840 con 0x7f959015e790 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f958e7fc640 1 -- 172.21.15.12:0/735067743 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580030c10 con 0x7f959015e790 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f9595ba8640 1 -- 172.21.15.12:0/735067743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f959015e790 msgr2=0x7f95900d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f9595ba8640 1 --2- 172.21.15.12:0/735067743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f959015e790 0x7f95900d3240 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f95900d2f30 tx=0x7f958002b980 comp rx=0 tx=0).stop 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f9595ba8640 1 -- 172.21.15.12:0/735067743 shutdown_connections 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f9595ba8640 1 --2- 172.21.15.12:0/735067743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f95900d1610 0x7f95900d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f9595ba8640 1 --2- 172.21.15.12:0/735067743 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f959015e790 0x7f95900d3240 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f9595ba8640 1 -- 172.21.15.12:0/735067743 >> 172.21.15.12:0/735067743 conn(0x7f959008bf60 msgr2=0x7f95900d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f9595ba8640 1 -- 172.21.15.12:0/735067743 shutdown_connections 2024-04-15T15:21:57.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.238+0000 7f9595ba8640 1 -- 172.21.15.12:0/735067743 wait complete. 2024-04-15T15:21:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.239+0000 7f9595ba8640 1 Processor -- start 2024-04-15T15:21:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f9595ba8640 1 -- start start 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f9595ba8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95900cfd20 0x7f95901f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f9595ba8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95900d1610 0x7f9590159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f9595ba8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f959015e790 0x7f959015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f9595ba8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f959015ecf0 con 0x7f95900d1610 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f9595ba8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f959015eb70 con 0x7f95900cfd20 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f9595ba8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f95900b74f0 con 0x7f959015e790 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f958effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95900d1610 0x7f9590159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f958effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95900d1610 0x7f9590159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52720/0 (socket says 172.21.15.12:52720) 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f958effd640 1 -- 172.21.15.12:0/442846287 learned_addr learned my addr 172.21.15.12:0/442846287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f958f7fe640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95900cfd20 0x7f95901f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.240+0000 7f958ffff640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f959015e790 0x7f959015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f958effd640 1 -- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f959015e790 msgr2=0x7f959015a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f958effd640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f959015e790 0x7f959015a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f958effd640 1 -- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95900cfd20 msgr2=0x7f95901f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f958effd640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95900cfd20 0x7f95901f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f958effd640 1 -- 172.21.15.12:0/442846287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959015acc0 con 0x7f95900d1610 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f958effd640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95900d1610 0x7f9590159ff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f957800c400 tx=0x7f957800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f9587fff640 1 -- 172.21.15.12:0/442846287 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95780052d0 con 0x7f95900d1610 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f9587fff640 1 -- 172.21.15.12:0/442846287 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9578005490 con 0x7f95900d1610 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f9587fff640 1 -- 172.21.15.12:0/442846287 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578007260 con 0x7f95900d1610 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959015afa0 con 0x7f95900d1610 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.241+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959015b560 con 0x7f95900d1610 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.242+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95900689e0 con 0x7f95900d1610 2024-04-15T15:21:57.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.242+0000 7f9587fff640 1 -- 172.21.15.12:0/442846287 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9578007400 con 0x7f95900d1610 2024-04-15T15:21:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.243+0000 7f9587fff640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f957006c8f0 0x7f957006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.243+0000 7f9587fff640 1 -- 172.21.15.12:0/442846287 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f957808b3e0 con 0x7f95900d1610 2024-04-15T15:21:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.243+0000 7f958f7fe640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f957006c8f0 0x7f957006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.244+0000 7f958f7fe640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f957006c8f0 0x7f957006edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9580036000 tx=0x7f958003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.247+0000 7f9587fff640 1 -- 172.21.15.12:0/442846287 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9578058e10 con 0x7f95900d1610 2024-04-15T15:21:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.432+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f95900d1ce0 con 0x7f95900d1610 2024-04-15T15:21:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.433+0000 7f9587fff640 1 -- 172.21.15.12:0/442846287 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f957805d110 con 0x7f95900d1610 2024-04-15T15:21:57.435 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f957006c8f0 msgr2=0x7f957006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f957006c8f0 0x7f957006edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9580036000 tx=0x7f958003b040 comp rx=0 tx=0).stop 2024-04-15T15:21:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95900d1610 msgr2=0x7f9590159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95900d1610 0x7f9590159ff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f957800c400 tx=0x7f957800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 shutdown_connections 2024-04-15T15:21:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f959015e790 0x7f959015a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95900d1610 0x7f9590159ff0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f957006c8f0 0x7f957006edb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.434+0000 7f9595ba8640 1 --2- 172.21.15.12:0/442846287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95900cfd20 0x7f95901f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.435+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 >> 172.21.15.12:0/442846287 conn(0x7f959008bf60 msgr2=0x7f95900cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.435+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 shutdown_connections 2024-04-15T15:21:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.435+0000 7f9595ba8640 1 -- 172.21.15.12:0/442846287 wait complete. 2024-04-15T15:21:57.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:57.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.553+0000 7ff01fd0d640 1 Processor -- start 2024-04-15T15:21:57.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.553+0000 7ff01fd0d640 1 -- start start 2024-04-15T15:21:57.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01fd0d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff01809a9d0 0x7ff018160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01fd0d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff0180701f0 0x7ff01815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01fd0d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff018070720 con 0x7ff01809adb0 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01fd0d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff01809ba90 con 0x7ff01809a9d0 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01fd0d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff018160070 con 0x7ff0180701f0 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01daa9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff01809a9d0 0x7ff018160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01daa9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff01809a9d0 0x7ff018160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52736/0 (socket says 172.21.15.12:52736) 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01daa9640 1 -- 172.21.15.12:0/2510729153 learned_addr learned my addr 172.21.15.12:0/2510729153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01d2a8640 1 --2- 172.21.15.12:0/2510729153 >> v2:172.21.15.55:3300/0 conn(0x7ff0180701f0 0x7ff01815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01daa9640 1 -- 172.21.15.12:0/2510729153 >> v2:172.21.15.55:3300/0 conn(0x7ff0180701f0 msgr2=0x7ff01815fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01daa9640 1 --2- 172.21.15.12:0/2510729153 >> v2:172.21.15.55:3300/0 conn(0x7ff0180701f0 0x7ff01815fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01daa9640 1 -- 172.21.15.12:0/2510729153 >> v1:172.21.15.55:6789/0 conn(0x7ff01809adb0 legacy=0x7ff0180d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.554+0000 7ff01daa9640 1 -- 172.21.15.12:0/2510729153 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff018070a90 con 0x7ff01809a9d0 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.555+0000 7ff01daa9640 1 --2- 172.21.15.12:0/2510729153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01809a9d0 0x7ff018160420 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff018072180 tx=0x7ff01402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00c28101c078cfd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.555+0000 7ff01caa7640 1 -- 172.21.15.12:0/2510729153 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014035020 con 0x7ff01809a9d0 2024-04-15T15:21:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.555+0000 7ff01caa7640 1 -- 172.21.15.12:0/2510729153 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff014031870 con 0x7ff01809a9d0 2024-04-15T15:21:57.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.555+0000 7ff01caa7640 1 -- 172.21.15.12:0/2510729153 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014031c40 con 0x7ff01809a9d0 2024-04-15T15:21:57.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.556+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2510729153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01809a9d0 msgr2=0x7ff018160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.556+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2510729153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01809a9d0 0x7ff018160420 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff018072180 tx=0x7ff01402b910 comp rx=0 tx=0).stop 2024-04-15T15:21:57.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.556+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2510729153 shutdown_connections 2024-04-15T15:21:57.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.556+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2510729153 >> v2:172.21.15.55:3300/0 conn(0x7ff0180701f0 0x7ff01815fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.556+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2510729153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01809a9d0 0x7ff018160420 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.556+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2510729153 >> 172.21.15.12:0/2510729153 conn(0x7ff0180d2460 msgr2=0x7ff0180d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.556+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2510729153 shutdown_connections 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.556+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2510729153 wait complete. 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.557+0000 7ff01fd0d640 1 Processor -- start 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.557+0000 7ff01fd0d640 1 -- start start 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01fd0d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0180701f0 0x7ff01815aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01fd0d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff01809a9d0 0x7ff018154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01fd0d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff01809adb0 0x7ff018153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01fd0d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff018071050 con 0x7ff0180701f0 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01fd0d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff0180d08c0 con 0x7ff01809a9d0 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01fd0d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff0180b1c60 con 0x7ff01809adb0 2024-04-15T15:21:57.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0180701f0 0x7ff01815aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0180701f0 0x7ff01815aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52746/0 (socket says 172.21.15.12:52746) 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 -- 172.21.15.12:0/2345255102 learned_addr learned my addr 172.21.15.12:0/2345255102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01d2a8640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff01809a9d0 0x7ff018154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 -- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff01809adb0 msgr2=0x7ff018153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff01809adb0 0x7ff018153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 -- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff01809a9d0 msgr2=0x7ff018154130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff01809a9d0 0x7ff018154130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 -- 172.21.15.12:0/2345255102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0181eac50 con 0x7ff0180701f0 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01daa9640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0180701f0 0x7ff01815aa10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff0140097f0 tx=0x7ff0140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff00e7fc640 1 -- 172.21.15.12:0/2345255102 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014040070 con 0x7ff0180701f0 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff00e7fc640 1 -- 172.21.15.12:0/2345255102 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff01403a420 con 0x7ff0180701f0 2024-04-15T15:21:57.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.558+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01815a250 con 0x7ff0180701f0 2024-04-15T15:21:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.559+0000 7ff00e7fc640 1 -- 172.21.15.12:0/2345255102 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014035040 con 0x7ff0180701f0 2024-04-15T15:21:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.559+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0181eaef0 con 0x7ff0180701f0 2024-04-15T15:21:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.560+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0180689e0 con 0x7ff0180701f0 2024-04-15T15:21:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.560+0000 7ff00e7fc640 1 -- 172.21.15.12:0/2345255102 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff01404b060 con 0x7ff0180701f0 2024-04-15T15:21:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.560+0000 7ff00e7fc640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feffc06c8f0 0x7feffc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.561+0000 7ff00e7fc640 1 -- 172.21.15.12:0/2345255102 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7ff0140adea0 con 0x7ff0180701f0 2024-04-15T15:21:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.561+0000 7ff01d2a8640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feffc06c8f0 0x7feffc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.561+0000 7ff01d2a8640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feffc06c8f0 0x7feffc06edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff0180a05a0 tx=0x7ff008008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.565+0000 7ff00e7fc640 1 -- 172.21.15.12:0/2345255102 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff01407be90 con 0x7ff0180701f0 2024-04-15T15:21:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.749+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff01815e810 con 0x7ff0180701f0 2024-04-15T15:21:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.750+0000 7ff00e7fc640 1 -- 172.21.15.12:0/2345255102 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7ff01407b790 con 0x7ff0180701f0 2024-04-15T15:21:57.751 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:57.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feffc06c8f0 msgr2=0x7feffc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feffc06c8f0 0x7feffc06edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff0180a05a0 tx=0x7ff008008040 comp rx=0 tx=0).stop 2024-04-15T15:21:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0180701f0 msgr2=0x7ff01815aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0180701f0 0x7ff01815aa10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff0140097f0 tx=0x7ff0140046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 shutdown_connections 2024-04-15T15:21:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff01809adb0 0x7ff018153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feffc06c8f0 0x7feffc06edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff01809a9d0 0x7ff018154130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 --2- 172.21.15.12:0/2345255102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0180701f0 0x7ff01815aa10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 >> 172.21.15.12:0/2345255102 conn(0x7ff0180d2460 msgr2=0x7ff01815d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:57.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 shutdown_connections 2024-04-15T15:21:57.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.752+0000 7ff01fd0d640 1 -- 172.21.15.12:0/2345255102 wait complete. 2024-04-15T15:21:57.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:57.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.870+0000 7fd4f4a28640 1 Processor -- start 2024-04-15T15:21:57.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.870+0000 7fd4f4a28640 1 -- start start 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4f4a28640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 0x7fd4f00d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4f4a28640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd4f00cfd20 0x7fd4f00d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4f4a28640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd4f00d0f00 0x7fd4f00d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4f4a28640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd4f00d2db0 con 0x7fd4f00d0f00 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4f4a28640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd4f00d2480 con 0x7fd4f00cfd20 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4f4a28640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f00d2760 con 0x7fd4f015e790 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4eed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd4f00d0f00 0x7fd4f00d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4eed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd4f00d0f00 0x7fd4f00d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52750/0 (socket says 172.21.15.12:52750) 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4eed9d640 1 -- 172.21.15.12:0/1730207729 learned_addr learned my addr 172.21.15.12:0/1730207729 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4ee59c640 1 --2- 172.21.15.12:0/1730207729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 0x7fd4f00d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4edd9b640 1 --2- 172.21.15.12:0/1730207729 >> v2:172.21.15.55:3300/0 conn(0x7fd4f00cfd20 0x7fd4f00d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4eed9d640 1 -- 172.21.15.12:0/1730207729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 msgr2=0x7fd4f00d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4eed9d640 1 --2- 172.21.15.12:0/1730207729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 0x7fd4f00d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4eed9d640 1 -- 172.21.15.12:0/1730207729 >> v2:172.21.15.55:3300/0 conn(0x7fd4f00cfd20 msgr2=0x7fd4f00d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4eed9d640 1 --2- 172.21.15.12:0/1730207729 >> v2:172.21.15.55:3300/0 conn(0x7fd4f00cfd20 0x7fd4f00d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.871+0000 7fd4eed9d640 1 -- 172.21.15.12:0/1730207729 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f00d2a40 con 0x7fd4f00d0f00 2024-04-15T15:21:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.872+0000 7fd4eed9d640 1 --2- 172.21.15.12:0/1730207729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00d13b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd4f008d020 tx=0x7fd4e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2557ad9ea93136a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.872+0000 7fd4ed59a640 1 -- 172.21.15.12:0/1730207729 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e802ca70 con 0x7fd4f00d0f00 2024-04-15T15:21:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.872+0000 7fd4ed59a640 1 -- 172.21.15.12:0/1730207729 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4e802cc10 con 0x7fd4f00d0f00 2024-04-15T15:21:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.872+0000 7fd4ed59a640 1 -- 172.21.15.12:0/1730207729 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e80343f0 con 0x7fd4f00d0f00 2024-04-15T15:21:57.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 -- 172.21.15.12:0/1730207729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 msgr2=0x7fd4f00d13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/1730207729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00d13b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd4f008d020 tx=0x7fd4e802bed0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 -- 172.21.15.12:0/1730207729 shutdown_connections 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/1730207729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00d13b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/1730207729 >> v2:172.21.15.55:3300/0 conn(0x7fd4f00cfd20 0x7fd4f00d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/1730207729 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 0x7fd4f00d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 -- 172.21.15.12:0/1730207729 >> 172.21.15.12:0/1730207729 conn(0x7fd4f008bf60 msgr2=0x7fd4f00d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 -- 172.21.15.12:0/1730207729 shutdown_connections 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.873+0000 7fd4f4a28640 1 -- 172.21.15.12:0/1730207729 wait complete. 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.874+0000 7fd4f4a28640 1 Processor -- start 2024-04-15T15:21:57.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.874+0000 7fd4f4a28640 1 -- start start 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4f4a28640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4f00cfd20 0x7fd4f00c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4f4a28640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4f4a28640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 0x7fd4f00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4f4a28640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f015ecf0 con 0x7fd4f00d0f00 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4f4a28640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f015eb70 con 0x7fd4f00cfd20 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4f4a28640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f00b74f0 con 0x7fd4f015e790 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52758/0 (socket says 172.21.15.12:52758) 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 -- 172.21.15.12:0/4202426266 learned_addr learned my addr 172.21.15.12:0/4202426266 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:57.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4eed9d640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 0x7fd4f00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4ee59c640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4f00cfd20 0x7fd4f00c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 -- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 msgr2=0x7fd4f00c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 0x7fd4f00c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 -- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4f00cfd20 msgr2=0x7fd4f00c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4f00cfd20 0x7fd4f00c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 -- 172.21.15.12:0/4202426266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f00c9be0 con 0x7fd4f00d0f00 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4ee59c640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4f00cfd20 0x7fd4f00c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.875+0000 7fd4edd9b640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00cce40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400d800 tx=0x7fd4e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.876+0000 7fd4deffd640 1 -- 172.21.15.12:0/4202426266 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4018020 con 0x7fd4f00d0f00 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.876+0000 7fd4deffd640 1 -- 172.21.15.12:0/4202426266 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4e40096a0 con 0x7fd4f00d0f00 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.876+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f00c9ec0 con 0x7fd4f00d0f00 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.876+0000 7fd4deffd640 1 -- 172.21.15.12:0/4202426266 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4009a70 con 0x7fd4f00d0f00 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.876+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f00ca480 con 0x7fd4f00d0f00 2024-04-15T15:21:57.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.877+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f00689e0 con 0x7fd4f00d0f00 2024-04-15T15:21:57.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.877+0000 7fd4deffd640 1 -- 172.21.15.12:0/4202426266 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd4e401b020 con 0x7fd4f00d0f00 2024-04-15T15:21:57.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.878+0000 7fd4deffd640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd4d006c8f0 0x7fd4d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:57.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.878+0000 7fd4deffd640 1 -- 172.21.15.12:0/4202426266 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fd4e408c160 con 0x7fd4f00d0f00 2024-04-15T15:21:57.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.878+0000 7fd4ee59c640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd4d006c8f0 0x7fd4d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:57.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.878+0000 7fd4ee59c640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd4d006c8f0 0x7fd4d006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0006fd0 tx=0x7fd4e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:57.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:57.882+0000 7fd4deffd640 1 -- 172.21.15.12:0/4202426266 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd4e4022030 con 0x7fd4f00d0f00 2024-04-15T15:21:58.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.067+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd4f00ce9d0 con 0x7fd4f00d0f00 2024-04-15T15:21:58.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.067+0000 7fd4deffd640 1 -- 172.21.15.12:0/4202426266 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fd4e405a150 con 0x7fd4f00d0f00 2024-04-15T15:21:58.069 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd4d006c8f0 msgr2=0x7fd4d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd4d006c8f0 0x7fd4d006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0006fd0 tx=0x7fd4e0008040 comp rx=0 tx=0).stop 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 msgr2=0x7fd4f00cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00cce40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400d800 tx=0x7fd4e400dcc0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 shutdown_connections 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd4f015e790 0x7fd4f00c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4f00d0f00 0x7fd4f00cce40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd4d006c8f0 0x7fd4d006edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 --2- 172.21.15.12:0/4202426266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4f00cfd20 0x7fd4f00c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 >> 172.21.15.12:0/4202426266 conn(0x7fd4f008bf60 msgr2=0x7fd4f00d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 shutdown_connections 2024-04-15T15:21:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.069+0000 7fd4f4a28640 1 -- 172.21.15.12:0/4202426266 wait complete. 2024-04-15T15:21:58.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:58.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.188+0000 7f26f5524640 1 Processor -- start 2024-04-15T15:21:58.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.188+0000 7f26f5524640 1 -- start start 2024-04-15T15:21:58.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.188+0000 7f26f5524640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f26f00d3240 0x7f26f00d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.188+0000 7f26f5524640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f00d2480 con 0x7f26f015e790 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.188+0000 7f26f5524640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f00d2760 con 0x7f26f00d1440 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.188+0000 7f26f5524640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f26f00d2db0 con 0x7f26f00d3240 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26eeffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f26f015e790 0x7f26f00cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33558/0 (socket says 172.21.15.12:33558) 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26eeffd640 1 -- 172.21.15.12:0/1797456586 learned_addr learned my addr 172.21.15.12:0/1797456586 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26ee7fc640 1 --2- 172.21.15.12:0/1797456586 >> v2:172.21.15.55:3300/0 conn(0x7f26f00d3240 0x7f26f00d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1136432559 0 0) 0x7f26f00d2480 con 0x7f26f015e790 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26dc003520 con 0x7f26f015e790 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 51998082 0 0) 0x7f26dc003520 con 0x7f26f015e790 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 >> v2:172.21.15.55:3300/0 conn(0x7f26f00d3240 msgr2=0x7f26f00d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 --2- 172.21.15.12:0/1797456586 >> v2:172.21.15.55:3300/0 conn(0x7f26f00d3240 0x7f26f00d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 >> v1:172.21.15.55:6789/0 conn(0x7f26f00d1440 legacy=0x7f26f00b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f00d2a40 con 0x7f26f015e790 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f26e00019f0 con 0x7f26f015e790 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 >> v1:172.21.15.12:6789/0 conn(0x7f26f015e790 legacy=0x7f26f00cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26dc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26dc004800 0x7f26dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26dc0051c0 0x7f26dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f26f00d2db0 con 0x7f26dc0039b0 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc00c090 con 0x7f26dc004800 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f26dc00c290 con 0x7f26dc0051c0 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26eeffd640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26dc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26ee7fc640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26dc004800 0x7f26dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.189+0000 7f26ef7fe640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26dc0051c0 0x7f26dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.190+0000 7f26eeffd640 1 -- 172.21.15.12:0/1797456586 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26dc0051c0 msgr2=0x7f26dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.190+0000 7f26eeffd640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26dc0051c0 0x7f26dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.190+0000 7f26eeffd640 1 -- 172.21.15.12:0/1797456586 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26dc004800 msgr2=0x7f26dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.190+0000 7f26eeffd640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26dc004800 0x7f26dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.190+0000 7f26eeffd640 1 -- 172.21.15.12:0/1797456586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc003520 con 0x7f26dc0039b0 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.190+0000 7f26ef7fe640 1 -- 172.21.15.12:0/1797456586 reap_dead start 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.190+0000 7f26eeffd640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26dc003df0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f26e000a840 tx=0x7f26dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9157667fc02cf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e000a460 con 0x7f26dc0039b0 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e0034650 con 0x7f26dc0039b0 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26edffb640 1 -- 172.21.15.12:0/1797456586 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0034a40 con 0x7f26dc0039b0 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26f5524640 1 -- 172.21.15.12:0/1797456586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 msgr2=0x7f26dc003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26f5524640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26dc003df0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f26e000a840 tx=0x7f26dc005830 comp rx=0 tx=0).stop 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26f5524640 1 -- 172.21.15.12:0/1797456586 shutdown_connections 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26f5524640 1 --2- 172.21.15.12:0/1797456586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26dc003df0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26f5524640 1 -- 172.21.15.12:0/1797456586 >> 172.21.15.12:0/1797456586 conn(0x7f26f008bf60 msgr2=0x7f26f00d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:58.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26f5524640 1 -- 172.21.15.12:0/1797456586 shutdown_connections 2024-04-15T15:21:58.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.191+0000 7f26f5524640 1 -- 172.21.15.12:0/1797456586 wait complete. 2024-04-15T15:21:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.192+0000 7f26f5524640 1 Processor -- start 2024-04-15T15:21:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.192+0000 7f26f5524640 1 -- start start 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26f5524640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26f00ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26f5524640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26f00c91d0 0x7f26f0150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26f5524640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26f00ccb60 0x7f26f015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26f5524640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f26f00d5300 con 0x7f26dc0039b0 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26f5524640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f26f00d5180 con 0x7f26f00ccb60 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26f5524640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f26f015eb70 con 0x7f26f00c91d0 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26f00ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26f00ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52780/0 (socket says 172.21.15.12:52780) 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26eeffd640 1 -- 172.21.15.12:0/992211791 learned_addr learned my addr 172.21.15.12:0/992211791 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26ef7fe640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26f00ccb60 0x7f26f015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.193+0000 7f26ee7fc640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26f00c91d0 0x7f26f0150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26eeffd640 1 -- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26f00c91d0 msgr2=0x7f26f0150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26eeffd640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26f00c91d0 0x7f26f0150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26eeffd640 1 -- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26f00ccb60 msgr2=0x7f26f015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26eeffd640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26f00ccb60 0x7f26f015b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26eeffd640 1 -- 172.21.15.12:0/992211791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f00c9820 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26ef7fe640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26f00ccb60 0x7f26f015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26eeffd640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26f00ca8c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f26e000a930 tx=0x7f26f00d2930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26e77fe640 1 -- 172.21.15.12:0/992211791 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0004d60 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26e77fe640 1 -- 172.21.15.12:0/992211791 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e0044070 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f00c9b00 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26e77fe640 1 -- 172.21.15.12:0/992211791 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0034a00 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.194+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f00ca0c0 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.195+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f00689e0 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.195+0000 7f26e77fe640 1 -- 172.21.15.12:0/992211791 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f26e0034ba0 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.196+0000 7f26e77fe640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f26d006c920 0x7f26d006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.196+0000 7f26e77fe640 1 -- 172.21.15.12:0/992211791 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f26e00ae660 con 0x7f26dc0039b0 2024-04-15T15:21:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.196+0000 7f26ee7fc640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f26d006c920 0x7f26d006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.197+0000 7f26ee7fc640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f26d006c920 0x7f26d006ede0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f26f01f7920 tx=0x7f26d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.200+0000 7f26e77fe640 1 -- 172.21.15.12:0/992211791 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26e003e380 con 0x7f26dc0039b0 2024-04-15T15:21:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.384+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f26f00d1ce0 con 0x7f26dc0039b0 2024-04-15T15:21:58.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.384+0000 7f26e77fe640 1 -- 172.21.15.12:0/992211791 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f26e007c650 con 0x7f26dc0039b0 2024-04-15T15:21:58.386 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f26d006c920 msgr2=0x7f26d006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f26d006c920 0x7f26d006ede0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f26f01f7920 tx=0x7f26d8008040 comp rx=0 tx=0).stop 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 msgr2=0x7f26f00ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26f00ca8c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f26e000a930 tx=0x7f26f00d2930 comp rx=0 tx=0).stop 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 shutdown_connections 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f26d006c920 0x7f26d006ede0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26f00ccb60 0x7f26f015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f26f00c91d0 0x7f26f0150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 --2- 172.21.15.12:0/992211791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26dc0039b0 0x7f26f00ca8c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 >> 172.21.15.12:0/992211791 conn(0x7f26f008bf60 msgr2=0x7f26f00d4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 shutdown_connections 2024-04-15T15:21:58.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.386+0000 7f26f5524640 1 -- 172.21.15.12:0/992211791 wait complete. 2024-04-15T15:21:58.410 INFO:tasks.cephfs_test_runner:Starting test: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:21:58.410 INFO:tasks.cephfs_test_runner:test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:21:58.410 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:21:58.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.513+0000 7f27878f2640 1 Processor -- start 2024-04-15T15:21:58.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.513+0000 7f27878f2640 1 -- start start 2024-04-15T15:21:58.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f27878f2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2780070940 0x7f278009b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f27878f2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 0x7f27800d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f27878f2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27800d05e0 con 0x7f2780160310 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f27878f2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f278015f720 con 0x7f2780070940 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f27878f2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f278015fa00 con 0x7f27800d0da0 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f278568e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2780070940 0x7f278009b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f2784e8d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2780160310 0x7f2780070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33566/0 (socket says 172.21.15.12:33566) 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f2784e8d640 1 -- 172.21.15.12:0/4175423756 learned_addr learned my addr 172.21.15.12:0/4175423756 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f2785e8f640 1 --2- 172.21.15.12:0/4175423756 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 0x7f27800d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f278568e640 1 -- 172.21.15.12:0/4175423756 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 msgr2=0x7f27800d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f278568e640 1 --2- 172.21.15.12:0/4175423756 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 0x7f27800d1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f278568e640 1 -- 172.21.15.12:0/4175423756 >> v1:172.21.15.12:6789/0 conn(0x7f2780160310 legacy=0x7f2780070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f2777fff640 1 -- 172.21.15.12:0/4175423756 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573170858 0 0) 0x7f27800d05e0 con 0x7f2780160310 2024-04-15T15:21:58.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.514+0000 7f278568e640 1 -- 172.21.15.12:0/4175423756 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278015fce0 con 0x7f2780070940 2024-04-15T15:21:58.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.515+0000 7f278568e640 1 --2- 172.21.15.12:0/4175423756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f278009b880 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097a0 tx=0x7f277c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fff7f5cfe300d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.515+0000 7f2777fff640 1 -- 172.21.15.12:0/4175423756 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c035020 con 0x7f2780070940 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.515+0000 7f2777fff640 1 -- 172.21.15.12:0/4175423756 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f277c031830 con 0x7f2780070940 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.515+0000 7f2777fff640 1 -- 172.21.15.12:0/4175423756 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c031c00 con 0x7f2780070940 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.516+0000 7f27878f2640 1 -- 172.21.15.12:0/4175423756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 msgr2=0x7f278009b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.516+0000 7f27878f2640 1 --2- 172.21.15.12:0/4175423756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f278009b880 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097a0 tx=0x7f277c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.516+0000 7f27878f2640 1 -- 172.21.15.12:0/4175423756 shutdown_connections 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.516+0000 7f27878f2640 1 --2- 172.21.15.12:0/4175423756 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 0x7f27800d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.516+0000 7f27878f2640 1 --2- 172.21.15.12:0/4175423756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f278009b880 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.516+0000 7f27878f2640 1 -- 172.21.15.12:0/4175423756 >> 172.21.15.12:0/4175423756 conn(0x7f27800b3f40 msgr2=0x7f27800b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.516+0000 7f27878f2640 1 -- 172.21.15.12:0/4175423756 shutdown_connections 2024-04-15T15:21:58.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.516+0000 7f27878f2640 1 -- 172.21.15.12:0/4175423756 wait complete. 2024-04-15T15:21:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.517+0000 7f27878f2640 1 Processor -- start 2024-04-15T15:21:58.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.517+0000 7f27878f2640 1 -- start start 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f27878f2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f2780157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f27878f2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 0x7f27800c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f27878f2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2780160310 0x7f27800c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f27878f2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f27800d1770 con 0x7f2780070940 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f27878f2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2780160770 con 0x7f2780160310 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f27878f2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f27800d18f0 con 0x7f27800d0da0 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f2780157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f2780157810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52810/0 (socket says 172.21.15.12:52810) 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 -- 172.21.15.12:0/2959649902 learned_addr learned my addr 172.21.15.12:0/2959649902 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 -- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 msgr2=0x7f27800c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f2785e8f640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2780160310 0x7f27800c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f2784e8d640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 0x7f27800c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 0x7f27800c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 -- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2780160310 msgr2=0x7f27800c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2780160310 0x7f27800c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 -- 172.21.15.12:0/2959649902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2780157d50 con 0x7f2780070940 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f2785e8f640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2780160310 0x7f27800c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f278568e640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f2780157810 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097a0 tx=0x7f277c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f2775ffb640 1 -- 172.21.15.12:0/2959649902 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c041070 con 0x7f2780070940 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f2775ffb640 1 -- 172.21.15.12:0/2959649902 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f277c004250 con 0x7f2780070940 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27801f49a0 con 0x7f2780070940 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.518+0000 7f2775ffb640 1 -- 172.21.15.12:0/2959649902 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c035040 con 0x7f2780070940 2024-04-15T15:21:58.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.519+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27801f4f60 con 0x7f2780070940 2024-04-15T15:21:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.520+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27800689e0 con 0x7f2780070940 2024-04-15T15:21:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.520+0000 7f2775ffb640 1 -- 172.21.15.12:0/2959649902 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f277c038020 con 0x7f2780070940 2024-04-15T15:21:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.521+0000 7f2775ffb640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f276406c8f0 0x7f276406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.521+0000 7f2775ffb640 1 -- 172.21.15.12:0/2959649902 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f277c0adaa0 con 0x7f2780070940 2024-04-15T15:21:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.521+0000 7f2784e8d640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f276406c8f0 0x7f276406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.521+0000 7f2784e8d640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f276406c8f0 0x7f276406edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2770007fd0 tx=0x7f2770009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.525+0000 7f2775ffb640 1 -- 172.21.15.12:0/2959649902 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f277c07ba90 con 0x7f2780070940 2024-04-15T15:21:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.714+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2780070720 con 0x7f2780070940 2024-04-15T15:21:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.715+0000 7f2775ffb640 1 -- 172.21.15.12:0/2959649902 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f277c07b390 con 0x7f2780070940 2024-04-15T15:21:58.716 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:21:58.717 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:21:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f276406c8f0 msgr2=0x7f276406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f276406c8f0 0x7f276406edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2770007fd0 tx=0x7f2770009040 comp rx=0 tx=0).stop 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 msgr2=0x7f2780157810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f2780157810 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097a0 tx=0x7f277c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 shutdown_connections 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f276406c8f0 0x7f276406edb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2780160310 0x7f27800c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f27800d0da0 0x7f27800c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 --2- 172.21.15.12:0/2959649902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2780070940 0x7f2780157810 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 >> 172.21.15.12:0/2959649902 conn(0x7f27800b3f40 msgr2=0x7f27800b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 shutdown_connections 2024-04-15T15:21:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.717+0000 7f27878f2640 1 -- 172.21.15.12:0/2959649902 wait complete. 2024-04-15T15:21:58.729 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:21:58.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:21:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.835+0000 7f701051c640 1 Processor -- start 2024-04-15T15:21:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.835+0000 7f701051c640 1 -- start start 2024-04-15T15:21:58.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f701051c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f70080d3240 0x7f70080d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f701051c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70080d1440 0x7f70080b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f701051c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70080d2480 con 0x7f700815e790 2024-04-15T15:21:58.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f701051c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f70080d2db0 con 0x7f70080d1440 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f701051c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f70080d2760 con 0x7f70080d3240 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f700eab9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f70080d1440 0x7f70080b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f700e2b8640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f700815e790 0x7f70080cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33568/0 (socket says 172.21.15.12:33568) 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f700e2b8640 1 -- 172.21.15.12:0/3094364528 learned_addr learned my addr 172.21.15.12:0/3094364528 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.836+0000 7f700dab7640 1 --2- 172.21.15.12:0/3094364528 >> v2:172.21.15.55:3300/0 conn(0x7f70080d3240 0x7f70080d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.837+0000 7f700eab9640 1 -- 172.21.15.12:0/3094364528 >> v2:172.21.15.55:3300/0 conn(0x7f70080d3240 msgr2=0x7f70080d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.837+0000 7f700eab9640 1 --2- 172.21.15.12:0/3094364528 >> v2:172.21.15.55:3300/0 conn(0x7f70080d3240 0x7f70080d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.837+0000 7f700eab9640 1 -- 172.21.15.12:0/3094364528 >> v1:172.21.15.12:6789/0 conn(0x7f700815e790 legacy=0x7f70080cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.837+0000 7f700eab9640 1 -- 172.21.15.12:0/3094364528 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70080d2a40 con 0x7f70080d1440 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.837+0000 7f700d2b6640 1 -- 172.21.15.12:0/3094364528 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946980293 0 0) 0x7f70080d2480 con 0x7f700815e790 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.837+0000 7f700dab7640 1 --2- 172.21.15.12:0/3094364528 >> v2:172.21.15.55:3300/0 conn(0x7f70080d3240 0x7f70080d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.837+0000 7f700eab9640 1 --2- 172.21.15.12:0/3094364528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d1440 0x7f70080b74f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f700808e510 tx=0x7f700002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be62b8a38d28eef server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f700d2b6640 1 -- 172.21.15.12:0/3094364528 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000035020 con 0x7f70080d1440 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f700d2b6640 1 -- 172.21.15.12:0/3094364528 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7000031830 con 0x7f70080d1440 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f700d2b6640 1 -- 172.21.15.12:0/3094364528 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000031c00 con 0x7f70080d1440 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f701051c640 1 -- 172.21.15.12:0/3094364528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d1440 msgr2=0x7f70080b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f701051c640 1 --2- 172.21.15.12:0/3094364528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d1440 0x7f70080b74f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f700808e510 tx=0x7f700002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f701051c640 1 -- 172.21.15.12:0/3094364528 shutdown_connections 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f701051c640 1 --2- 172.21.15.12:0/3094364528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d1440 0x7f70080b74f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f701051c640 1 --2- 172.21.15.12:0/3094364528 >> v2:172.21.15.55:3300/0 conn(0x7f70080d3240 0x7f70080d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f701051c640 1 -- 172.21.15.12:0/3094364528 >> 172.21.15.12:0/3094364528 conn(0x7f700808bf60 msgr2=0x7f70080d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f701051c640 1 -- 172.21.15.12:0/3094364528 shutdown_connections 2024-04-15T15:21:58.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.838+0000 7f701051c640 1 -- 172.21.15.12:0/3094364528 wait complete. 2024-04-15T15:21:58.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.839+0000 7f701051c640 1 Processor -- start 2024-04-15T15:21:58.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.839+0000 7f701051c640 1 -- start start 2024-04-15T15:21:58.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f701051c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f70080d1440 0x7f7008156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f701051c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d3240 0x7f700809f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f701051c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f700815e790 0x7f70080cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f701051c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f70080d5180 con 0x7f70080d3240 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f701051c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f70080d0430 con 0x7f700815e790 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f701051c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f70080d5300 con 0x7f70080d1440 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f700dab7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d3240 0x7f700809f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f700dab7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d3240 0x7f700809f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52822/0 (socket says 172.21.15.12:52822) 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.840+0000 7f700dab7640 1 -- 172.21.15.12:0/4083452938 learned_addr learned my addr 172.21.15.12:0/4083452938 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f700dab7640 1 -- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f70080d1440 msgr2=0x7f7008156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f700dab7640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f70080d1440 0x7f7008156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f700dab7640 1 -- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f700815e790 msgr2=0x7f70080cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f700eab9640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f700815e790 0x7f70080cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f700dab7640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f700815e790 0x7f70080cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f700dab7640 1 -- 172.21.15.12:0/4083452938 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700809f940 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f700eab9640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f700815e790 0x7f70080cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f700dab7640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d3240 0x7f700809f040 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00c7f0 tx=0x7f6ffc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f7006ffd640 1 -- 172.21.15.12:0/4083452938 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc015070 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f7006ffd640 1 -- 172.21.15.12:0/4083452938 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ffc007a30 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f700809fc20 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f7006ffd640 1 -- 172.21.15.12:0/4083452938 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc007e00 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.841+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70080a04c0 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.842+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70080689e0 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.842+0000 7f7006ffd640 1 -- 172.21.15.12:0/4083452938 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ffc018070 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.843+0000 7f7006ffd640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6ff406c8f0 0x7f6ff406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.843+0000 7f7006ffd640 1 -- 172.21.15.12:0/4083452938 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f6ffc08bfa0 con 0x7f70080d3240 2024-04-15T15:21:58.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.843+0000 7f700e2b8640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6ff406c8f0 0x7f6ff406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:58.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.844+0000 7f700e2b8640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6ff406c8f0 0x7f6ff406edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8007fd0 tx=0x7f6ff8007ce0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:58.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:58.847+0000 7f7006ffd640 1 -- 172.21.15.12:0/4083452938 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ffc059f90 con 0x7f70080d3240 2024-04-15T15:21:59.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.031+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f70080d1ce0 con 0x7f70080d3240 2024-04-15T15:21:59.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.031+0000 7f7006ffd640 1 -- 172.21.15.12:0/4083452938 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f6ffc059890 con 0x7f70080d3240 2024-04-15T15:21:59.033 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:21:59.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6ff406c8f0 msgr2=0x7f6ff406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6ff406c8f0 0x7f6ff406edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8007fd0 tx=0x7f6ff8007ce0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d3240 msgr2=0x7f700809f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d3240 0x7f700809f040 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00c7f0 tx=0x7f6ffc00cbf0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 shutdown_connections 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6ff406c8f0 0x7f6ff406edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f700815e790 0x7f70080cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70080d3240 0x7f700809f040 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 --2- 172.21.15.12:0/4083452938 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f70080d1440 0x7f7008156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 >> 172.21.15.12:0/4083452938 conn(0x7f700808bf60 msgr2=0x7f700806cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 shutdown_connections 2024-04-15T15:21:59.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.034+0000 7f701051c640 1 -- 172.21.15.12:0/4083452938 wait complete. 2024-04-15T15:21:59.047 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:21:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.153+0000 7fb562428640 1 Processor -- start 2024-04-15T15:21:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.153+0000 7fb562428640 1 -- start start 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb562428640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c15e7a0 0x7fb55c0d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb562428640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb55c0d13f0 0x7fb55c162370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb562428640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb55c0d0530 con 0x7fb55c0d1870 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb562428640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb55c0cf490 con 0x7fb55c0d13f0 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb562428640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb55c0cf770 con 0x7fb55c15e7a0 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb55bfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c15e7a0 0x7fb55c0d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb553fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb55c0d13f0 0x7fb55c162370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb5609c5640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fb55c0d1870 0x7fb55c1628b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:56118/0 (socket says 172.21.15.12:56118) 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.154+0000 7fb5609c5640 1 -- 172.21.15.12:0/1726104014 learned_addr learned my addr 172.21.15.12:0/1726104014 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.155+0000 7fb55bfff640 1 -- 172.21.15.12:0/1726104014 >> v2:172.21.15.55:3300/0 conn(0x7fb55c0d13f0 msgr2=0x7fb55c162370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.155+0000 7fb55bfff640 1 --2- 172.21.15.12:0/1726104014 >> v2:172.21.15.55:3300/0 conn(0x7fb55c0d13f0 0x7fb55c162370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.155+0000 7fb55bfff640 1 -- 172.21.15.12:0/1726104014 >> v1:172.21.15.55:6789/0 conn(0x7fb55c0d1870 legacy=0x7fb55c1628b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.155+0000 7fb55bfff640 1 -- 172.21.15.12:0/1726104014 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55c0cfa50 con 0x7fb55c15e7a0 2024-04-15T15:21:59.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.155+0000 7fb553fff640 1 --2- 172.21.15.12:0/1726104014 >> v2:172.21.15.55:3300/0 conn(0x7fb55c0d13f0 0x7fb55c162370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.155+0000 7fb55bfff640 1 --2- 172.21.15.12:0/1726104014 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c15e7a0 0x7fb55c0d0eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0097a0 tx=0x7fb54c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e241672fdeb8dc72 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb55b7fe640 1 -- 172.21.15.12:0/1726104014 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c031780 con 0x7fb55c15e7a0 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb55b7fe640 1 -- 172.21.15.12:0/1726104014 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb54c037070 con 0x7fb55c15e7a0 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb55b7fe640 1 -- 172.21.15.12:0/1726104014 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c031b40 con 0x7fb55c15e7a0 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb562428640 1 -- 172.21.15.12:0/1726104014 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c15e7a0 msgr2=0x7fb55c0d0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb562428640 1 --2- 172.21.15.12:0/1726104014 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c15e7a0 0x7fb55c0d0eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0097a0 tx=0x7fb54c02b820 comp rx=0 tx=0).stop 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb562428640 1 -- 172.21.15.12:0/1726104014 shutdown_connections 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb562428640 1 --2- 172.21.15.12:0/1726104014 >> v2:172.21.15.55:3300/0 conn(0x7fb55c0d13f0 0x7fb55c162370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb562428640 1 --2- 172.21.15.12:0/1726104014 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c15e7a0 0x7fb55c0d0eb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb562428640 1 -- 172.21.15.12:0/1726104014 >> 172.21.15.12:0/1726104014 conn(0x7fb55c08bf60 msgr2=0x7fb55c0d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:59.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb562428640 1 -- 172.21.15.12:0/1726104014 shutdown_connections 2024-04-15T15:21:59.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.156+0000 7fb562428640 1 -- 172.21.15.12:0/1726104014 wait complete. 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.157+0000 7fb562428640 1 Processor -- start 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.157+0000 7fb562428640 1 -- start start 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb562428640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c0d13f0 0x7fb55c153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb562428640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55c0d1870 0x7fb55c160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb562428640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb55c15e7a0 0x7fb55c0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb562428640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c06db20 con 0x7fb55c0d1870 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb562428640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c0cf180 con 0x7fb55c15e7a0 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb562428640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb55c15eb80 con 0x7fb55c0d13f0 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb553fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55c0d1870 0x7fb55c160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb553fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55c0d1870 0x7fb55c160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52836/0 (socket says 172.21.15.12:52836) 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb553fff640 1 -- 172.21.15.12:0/2853901835 learned_addr learned my addr 172.21.15.12:0/2853901835 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb5609c5640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb55c15e7a0 0x7fb55c0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb55bfff640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c0d13f0 0x7fb55c153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb553fff640 1 -- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c0d13f0 msgr2=0x7fb55c153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb553fff640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c0d13f0 0x7fb55c153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb553fff640 1 -- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb55c15e7a0 msgr2=0x7fb55c0b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb553fff640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb55c15e7a0 0x7fb55c0b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb553fff640 1 -- 172.21.15.12:0/2853901835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55c161120 con 0x7fb55c0d1870 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.158+0000 7fb55bfff640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c0d13f0 0x7fb55c153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.159+0000 7fb553fff640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55c0d1870 0x7fb55c160730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb54400e960 tx=0x7fb54400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.159+0000 7fb5597fa640 1 -- 172.21.15.12:0/2853901835 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544017070 con 0x7fb55c0d1870 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.159+0000 7fb5597fa640 1 -- 172.21.15.12:0/2853901835 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb54400ca00 con 0x7fb55c0d1870 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.159+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb55c161400 con 0x7fb55c0d1870 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.159+0000 7fb5597fa640 1 -- 172.21.15.12:0/2853901835 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54400cdd0 con 0x7fb55c0d1870 2024-04-15T15:21:59.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.159+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb55c0b7350 con 0x7fb55c0d1870 2024-04-15T15:21:59.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.160+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb55c0b6300 con 0x7fb55c0d1870 2024-04-15T15:21:59.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.160+0000 7fb5597fa640 1 -- 172.21.15.12:0/2853901835 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb54401a070 con 0x7fb55c0d1870 2024-04-15T15:21:59.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.161+0000 7fb5597fa640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb53c06c8f0 0x7fb53c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.161+0000 7fb5597fa640 1 -- 172.21.15.12:0/2853901835 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fb54408be80 con 0x7fb55c0d1870 2024-04-15T15:21:59.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.161+0000 7fb55bfff640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb53c06c8f0 0x7fb53c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.162+0000 7fb55bfff640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb53c06c8f0 0x7fb53c06edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb54c036000 tx=0x7fb54c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.165+0000 7fb5597fa640 1 -- 172.21.15.12:0/2853901835 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb55c0b6300 con 0x7fb55c0d1870 2024-04-15T15:21:59.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.352+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb55c0b7140 con 0x7fb55c0d1870 2024-04-15T15:21:59.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.353+0000 7fb5597fa640 1 -- 172.21.15.12:0/2853901835 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fb544059e60 con 0x7fb55c0d1870 2024-04-15T15:21:59.355 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb53c06c8f0 msgr2=0x7fb53c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb53c06c8f0 0x7fb53c06edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb54c036000 tx=0x7fb54c008030 comp rx=0 tx=0).stop 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55c0d1870 msgr2=0x7fb55c160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55c0d1870 0x7fb55c160730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb54400e960 tx=0x7fb54400ee20 comp rx=0 tx=0).stop 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 shutdown_connections 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb53c06c8f0 0x7fb53c06edb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb55c15e7a0 0x7fb55c0b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb55c0d1870 0x7fb55c160730 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 --2- 172.21.15.12:0/2853901835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb55c0d13f0 0x7fb55c153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 >> 172.21.15.12:0/2853901835 conn(0x7fb55c08bf60 msgr2=0x7fb55c06cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 shutdown_connections 2024-04-15T15:21:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.355+0000 7fb562428640 1 -- 172.21.15.12:0/2853901835 wait complete. 2024-04-15T15:21:59.368 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:21:59.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.473+0000 7fceb2fd4640 1 Processor -- start 2024-04-15T15:21:59.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.474+0000 7fceb2fd4640 1 -- start start 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.474+0000 7fceb2fd4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fceac15e700 0x7fceac0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.474+0000 7fceb2fd4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fceac0d2360 0x7fceac0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.474+0000 7fceb2fd4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceac0b47a0 con 0x7fceac15eae0 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.474+0000 7fceb2fd4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fceac0b53b0 con 0x7fceac0d2360 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.474+0000 7fceb2fd4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fceac0b4a80 con 0x7fceac15e700 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceabfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fceac0d2360 0x7fceac0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceb1571640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fceac15eae0 0x7fceac0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33582/0 (socket says 172.21.15.12:33582) 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceb1571640 1 -- 172.21.15.12:0/1592757172 learned_addr learned my addr 172.21.15.12:0/1592757172 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceb0d70640 1 --2- 172.21.15.12:0/1592757172 >> v2:172.21.15.55:3300/0 conn(0x7fceac15e700 0x7fceac0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceabfff640 1 -- 172.21.15.12:0/1592757172 >> v2:172.21.15.55:3300/0 conn(0x7fceac15e700 msgr2=0x7fceac0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceabfff640 1 --2- 172.21.15.12:0/1592757172 >> v2:172.21.15.55:3300/0 conn(0x7fceac15e700 0x7fceac0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceabfff640 1 -- 172.21.15.12:0/1592757172 >> v1:172.21.15.12:6789/0 conn(0x7fceac15eae0 legacy=0x7fceac0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceab7fe640 1 -- 172.21.15.12:0/1592757172 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 58470492 0 0) 0x7fceac0b47a0 con 0x7fceac15eae0 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceabfff640 1 -- 172.21.15.12:0/1592757172 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceac0b4d60 con 0x7fceac0d2360 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.475+0000 7fceb0d70640 1 --2- 172.21.15.12:0/1592757172 >> v2:172.21.15.55:3300/0 conn(0x7fceac15e700 0x7fceac0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceabfff640 1 --2- 172.21.15.12:0/1592757172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0d2b60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fceac06c6c0 tx=0x7fce9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4065cf62b68ea2a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceab7fe640 1 -- 172.21.15.12:0/1592757172 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce94035020 con 0x7fceac0d2360 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceab7fe640 1 -- 172.21.15.12:0/1592757172 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce94031830 con 0x7fceac0d2360 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceab7fe640 1 -- 172.21.15.12:0/1592757172 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce94031c00 con 0x7fceac0d2360 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceb2fd4640 1 -- 172.21.15.12:0/1592757172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 msgr2=0x7fceac0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/1592757172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0d2b60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fceac06c6c0 tx=0x7fce9402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceb2fd4640 1 -- 172.21.15.12:0/1592757172 shutdown_connections 2024-04-15T15:21:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/1592757172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0d2b60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/1592757172 >> v2:172.21.15.55:3300/0 conn(0x7fceac15e700 0x7fceac0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceb2fd4640 1 -- 172.21.15.12:0/1592757172 >> 172.21.15.12:0/1592757172 conn(0x7fceac08bf60 msgr2=0x7fceac0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceb2fd4640 1 -- 172.21.15.12:0/1592757172 shutdown_connections 2024-04-15T15:21:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.476+0000 7fceb2fd4640 1 -- 172.21.15.12:0/1592757172 wait complete. 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.477+0000 7fceb2fd4640 1 Processor -- start 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb2fd4640 1 -- start start 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb2fd4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb2fd4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fceac15e700 0x7fceac1f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb2fd4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fceac15eae0 0x7fceac160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb2fd4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fceac0b5d80 con 0x7fceac0d2360 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb2fd4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fceac0b4450 con 0x7fceac15e700 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb2fd4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fceac0d1d40 con 0x7fceac15eae0 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb0d70640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.478+0000 7fceb0d70640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0c81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52856/0 (socket says 172.21.15.12:52856) 2024-04-15T15:21:59.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb0d70640 1 -- 172.21.15.12:0/2010297941 learned_addr learned my addr 172.21.15.12:0/2010297941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb0d70640 1 -- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fceac15eae0 msgr2=0x7fceac160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb0d70640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fceac15eae0 0x7fceac160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb0d70640 1 -- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fceac15e700 msgr2=0x7fceac1f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb0d70640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fceac15e700 0x7fceac1f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb0d70640 1 -- 172.21.15.12:0/2010297941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceac0b75e0 con 0x7fceac0d2360 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb0d70640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0c81a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcea400e810 tx=0x7fcea400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fcea97fa640 1 -- 172.21.15.12:0/2010297941 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea400f040 con 0x7fceac0d2360 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fcea97fa640 1 -- 172.21.15.12:0/2010297941 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea4004030 con 0x7fceac0d2360 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceac0b78c0 con 0x7fceac0d2360 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fcea97fa640 1 -- 172.21.15.12:0/2010297941 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea4004420 con 0x7fceac0d2360 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.479+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceac0b7e80 con 0x7fceac0d2360 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.480+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceac0ba290 con 0x7fceac0d2360 2024-04-15T15:21:59.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.481+0000 7fcea97fa640 1 -- 172.21.15.12:0/2010297941 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcea4005990 con 0x7fceac0d2360 2024-04-15T15:21:59.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.481+0000 7fcea97fa640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fce8c06c8f0 0x7fce8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.481+0000 7fcea97fa640 1 -- 172.21.15.12:0/2010297941 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fcea408c070 con 0x7fceac0d2360 2024-04-15T15:21:59.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.481+0000 7fceabfff640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fce8c06c8f0 0x7fce8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.482+0000 7fceabfff640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fce8c06c8f0 0x7fce8c06edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fce94036000 tx=0x7fce94030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.485+0000 7fcea97fa640 1 -- 172.21.15.12:0/2010297941 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcea4020020 con 0x7fceac0d2360 2024-04-15T15:21:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.670+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fceac0689e0 con 0x7fceac0d2360 2024-04-15T15:21:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.670+0000 7fcea97fa640 1 -- 172.21.15.12:0/2010297941 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fcea4017030 con 0x7fceac0d2360 2024-04-15T15:21:59.672 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fce8c06c8f0 msgr2=0x7fce8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fce8c06c8f0 0x7fce8c06edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fce94036000 tx=0x7fce94030000 comp rx=0 tx=0).stop 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 msgr2=0x7fceac0c81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0c81a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcea400e810 tx=0x7fcea400ecd0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 shutdown_connections 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fceac15eae0 0x7fceac160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fce8c06c8f0 0x7fce8c06edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fceac15e700 0x7fceac1f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 --2- 172.21.15.12:0/2010297941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceac0d2360 0x7fceac0c81a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 >> 172.21.15.12:0/2010297941 conn(0x7fceac08bf60 msgr2=0x7fceac0d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 shutdown_connections 2024-04-15T15:21:59.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.672+0000 7fceb2fd4640 1 -- 172.21.15.12:0/2010297941 wait complete. 2024-04-15T15:21:59.685 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:21:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.790+0000 7f213e59d640 1 Processor -- start 2024-04-15T15:21:59.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.790+0000 7f213e59d640 1 -- start start 2024-04-15T15:21:59.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.791+0000 7f213e59d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f213815e700 0x7f21380b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.791+0000 7f213e59d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21380b47a0 con 0x7f21380d2360 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.791+0000 7f213e59d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21380b4a80 con 0x7f21380d3270 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.791+0000 7f213e59d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f21380b53b0 con 0x7f213815e700 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.791+0000 7f21377fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f21380d2360 0x7f21380d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33596/0 (socket says 172.21.15.12:33596) 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.791+0000 7f21377fe640 1 -- 172.21.15.12:0/1941442660 learned_addr learned my addr 172.21.15.12:0/1941442660 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2137fff640 1 --2- 172.21.15.12:0/1941442660 >> v2:172.21.15.55:3300/0 conn(0x7f213815e700 0x7f21380b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 365591354 0 0) 0x7f21380b47a0 con 0x7f21380d2360 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2120003520 con 0x7f21380d2360 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1454650681 0 0) 0x7f2120003520 con 0x7f21380d2360 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 >> v2:172.21.15.55:3300/0 conn(0x7f213815e700 msgr2=0x7f21380b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 --2- 172.21.15.12:0/1941442660 >> v2:172.21.15.55:3300/0 conn(0x7f213815e700 0x7f21380b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 >> v1:172.21.15.55:6789/0 conn(0x7f21380d3270 legacy=0x7f21380cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21380b4d60 con 0x7f21380d2360 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f212c001670 con 0x7f21380d2360 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 >> v1:172.21.15.12:6789/0 conn(0x7f21380d2360 legacy=0x7f21380d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f2120003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2120004800 0x7f2120004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21200051c0 0x7f212000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f21380b53b0 con 0x7f21200039b0 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f212000c090 con 0x7f21200051c0 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f212000c290 con 0x7f2120004800 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2137fff640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f2120003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f213cb3a640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21200051c0 0x7f212000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f21377fe640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2120004800 0x7f2120004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2137fff640 1 -- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2120004800 msgr2=0x7f2120004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2137fff640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2120004800 0x7f2120004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2137fff640 1 -- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21200051c0 msgr2=0x7f212000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2137fff640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21200051c0 0x7f212000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f2137fff640 1 -- 172.21.15.12:0/1941442660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2120003520 con 0x7f21200039b0 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f213cb3a640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21200051c0 0x7f212000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.792+0000 7f213cb3a640 1 -- 172.21.15.12:0/1941442660 reap_dead start 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.793+0000 7f2137fff640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f2120003df0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2128009ab0 tx=0x7f2120005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3f95d30a0cf7a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.793+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2128009680 con 0x7f21200039b0 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.793+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2128005990 con 0x7f21200039b0 2024-04-15T15:21:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.793+0000 7f2136ffd640 1 -- 172.21.15.12:0/1941442660 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2128005d80 con 0x7f21200039b0 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.794+0000 7f213e59d640 1 -- 172.21.15.12:0/1941442660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 msgr2=0x7f2120003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.794+0000 7f213e59d640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f2120003df0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2128009ab0 tx=0x7f2120005830 comp rx=0 tx=0).stop 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.794+0000 7f213e59d640 1 -- 172.21.15.12:0/1941442660 shutdown_connections 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.794+0000 7f213e59d640 1 --2- 172.21.15.12:0/1941442660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f2120003df0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.794+0000 7f213e59d640 1 -- 172.21.15.12:0/1941442660 >> 172.21.15.12:0/1941442660 conn(0x7f213808bf60 msgr2=0x7f21380d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.794+0000 7f213e59d640 1 -- 172.21.15.12:0/1941442660 shutdown_connections 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.794+0000 7f213e59d640 1 -- 172.21.15.12:0/1941442660 wait complete. 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.795+0000 7f213e59d640 1 Processor -- start 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.795+0000 7f213e59d640 1 -- start start 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f213e59d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f21380b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f213e59d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21381529e0 0x7f21380cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f213e59d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f21380cded0 0x7f213815cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f213e59d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f21380d1d40 con 0x7f21200039b0 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f213e59d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f21380b4450 con 0x7f21381529e0 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f213e59d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f21380d1ec0 con 0x7f21380cded0 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f2137fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f21380b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f2137fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f21380b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52878/0 (socket says 172.21.15.12:52878) 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f2137fff640 1 -- 172.21.15.12:0/222462145 learned_addr learned my addr 172.21.15.12:0/222462145 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f21377fe640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21381529e0 0x7f21380cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f213cb3a640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f21380cded0 0x7f213815cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f2137fff640 1 -- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f21380cded0 msgr2=0x7f213815cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f2137fff640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f21380cded0 0x7f213815cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f2137fff640 1 -- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21381529e0 msgr2=0x7f21380cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f2137fff640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21381529e0 0x7f21380cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f2137fff640 1 -- 172.21.15.12:0/222462145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21381f8390 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.796+0000 7f213cb3a640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f21380cded0 0x7f213815cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.797+0000 7f2137fff640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f21380b7b60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2128009ba0 tx=0x7f2128004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.797+0000 7f2134ff9640 1 -- 172.21.15.12:0/222462145 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212802cd00 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.797+0000 7f2134ff9640 1 -- 172.21.15.12:0/222462145 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2128004030 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.797+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21381f8670 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.797+0000 7f2134ff9640 1 -- 172.21.15.12:0/222462145 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2128004420 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.797+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21381f8c30 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.798+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21381f8f70 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.798+0000 7f2134ff9640 1 -- 172.21.15.12:0/222462145 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f212803d040 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.799+0000 7f2134ff9640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f212406c8f0 0x7f212406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.799+0000 7f2134ff9640 1 -- 172.21.15.12:0/222462145 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f21280add40 con 0x7f21200039b0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.799+0000 7f21377fe640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f212406c8f0 0x7f212406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:21:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.800+0000 7f21377fe640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f212406c8f0 0x7f212406edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f212c001860 tx=0x7f212c007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:21:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.803+0000 7f2134ff9640 1 -- 172.21.15.12:0/222462145 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f21381f8f70 con 0x7f21200039b0 2024-04-15T15:21:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.988+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f21380689e0 con 0x7f21200039b0 2024-04-15T15:21:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.988+0000 7f2134ff9640 1 -- 172.21.15.12:0/222462145 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f2128043020 con 0x7f21200039b0 2024-04-15T15:21:59.990 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:21:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f212406c8f0 msgr2=0x7f212406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f212406c8f0 0x7f212406edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f212c001860 tx=0x7f212c007040 comp rx=0 tx=0).stop 2024-04-15T15:21:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 msgr2=0x7f21380b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:21:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f21380b7b60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2128009ba0 tx=0x7f2128004730 comp rx=0 tx=0).stop 2024-04-15T15:21:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 shutdown_connections 2024-04-15T15:21:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f21380cded0 0x7f213815cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f212406c8f0 0x7f212406edb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21381529e0 0x7f21380cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 --2- 172.21.15.12:0/222462145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21200039b0 0x7f21380b7b60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:21:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 >> 172.21.15.12:0/222462145 conn(0x7f213808bf60 msgr2=0x7f213806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:21:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 shutdown_connections 2024-04-15T15:21:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:21:59.990+0000 7f213e59d640 1 -- 172.21.15.12:0/222462145 wait complete. 2024-04-15T15:22:00.003 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-15T15:22:00.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.106+0000 7fb3fea63640 1 Processor -- start 2024-04-15T15:22:00.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.106+0000 7fb3fea63640 1 -- start start 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.107+0000 7fb3fea63640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb3f815e700 0x7fb3f80b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.107+0000 7fb3fea63640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb3f80d2360 0x7fb3f80d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.107+0000 7fb3fea63640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3f80b47a0 con 0x7fb3f815eae0 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.107+0000 7fb3fea63640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb3f80b53b0 con 0x7fb3f80d2360 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.107+0000 7fb3fea63640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb3f80b4a80 con 0x7fb3f815e700 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.107+0000 7fb3f77fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb3f80d2360 0x7fb3f80d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.107+0000 7fb3fd000640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb3f815eae0 0x7fb3f80cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33606/0 (socket says 172.21.15.12:33606) 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.107+0000 7fb3fd000640 1 -- 172.21.15.12:0/2898827039 learned_addr learned my addr 172.21.15.12:0/2898827039 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.108+0000 7fb3f7fff640 1 --2- 172.21.15.12:0/2898827039 >> v2:172.21.15.55:3300/0 conn(0x7fb3f815e700 0x7fb3f80b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.108+0000 7fb3f77fe640 1 -- 172.21.15.12:0/2898827039 >> v2:172.21.15.55:3300/0 conn(0x7fb3f815e700 msgr2=0x7fb3f80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.108+0000 7fb3f77fe640 1 --2- 172.21.15.12:0/2898827039 >> v2:172.21.15.55:3300/0 conn(0x7fb3f815e700 0x7fb3f80b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.108+0000 7fb3f77fe640 1 -- 172.21.15.12:0/2898827039 >> v1:172.21.15.12:6789/0 conn(0x7fb3f815eae0 legacy=0x7fb3f80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.108+0000 7fb3f77fe640 1 -- 172.21.15.12:0/2898827039 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f80b4d60 con 0x7fb3f80d2360 2024-04-15T15:22:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.108+0000 7fb3f6ffd640 1 -- 172.21.15.12:0/2898827039 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4241679841 0 0) 0x7fb3f80b47a0 con 0x7fb3f815eae0 2024-04-15T15:22:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.108+0000 7fb3f77fe640 1 --2- 172.21.15.12:0/2898827039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f80d2360 0x7fb3f80d2b60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb3f806c6c0 tx=0x7fb3ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6181a68987b42c82 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3f6ffd640 1 -- 172.21.15.12:0/2898827039 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec035020 con 0x7fb3f80d2360 2024-04-15T15:22:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3f6ffd640 1 -- 172.21.15.12:0/2898827039 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3ec031830 con 0x7fb3f80d2360 2024-04-15T15:22:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3f6ffd640 1 -- 172.21.15.12:0/2898827039 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec031c00 con 0x7fb3f80d2360 2024-04-15T15:22:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3fea63640 1 -- 172.21.15.12:0/2898827039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f80d2360 msgr2=0x7fb3f80d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3fea63640 1 --2- 172.21.15.12:0/2898827039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f80d2360 0x7fb3f80d2b60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb3f806c6c0 tx=0x7fb3ec02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3fea63640 1 -- 172.21.15.12:0/2898827039 shutdown_connections 2024-04-15T15:22:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3fea63640 1 --2- 172.21.15.12:0/2898827039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f80d2360 0x7fb3f80d2b60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3fea63640 1 --2- 172.21.15.12:0/2898827039 >> v2:172.21.15.55:3300/0 conn(0x7fb3f815e700 0x7fb3f80b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3fea63640 1 -- 172.21.15.12:0/2898827039 >> 172.21.15.12:0/2898827039 conn(0x7fb3f808bf60 msgr2=0x7fb3f80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3fea63640 1 -- 172.21.15.12:0/2898827039 shutdown_connections 2024-04-15T15:22:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.109+0000 7fb3fea63640 1 -- 172.21.15.12:0/2898827039 wait complete. 2024-04-15T15:22:00.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.110+0000 7fb3fea63640 1 Processor -- start 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.111+0000 7fb3fea63640 1 -- start start 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.111+0000 7fb3fea63640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3f80d2360 0x7fb3f81616f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.111+0000 7fb3fea63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f815e700 0x7fb3f814a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.111+0000 7fb3fea63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3f815eae0 0x7fb3f814a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.111+0000 7fb3fea63640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f80b5d80 con 0x7fb3f815e700 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.111+0000 7fb3fea63640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f80b4450 con 0x7fb3f80d2360 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.111+0000 7fb3fea63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb3f80d1d40 con 0x7fb3f815eae0 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f815e700 0x7fb3f814a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f815e700 0x7fb3f814a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52890/0 (socket says 172.21.15.12:52890) 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 -- 172.21.15.12:0/3380907768 learned_addr learned my addr 172.21.15.12:0/3380907768 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f7fff640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3f80d2360 0x7fb3f81616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3fd000640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3f815eae0 0x7fb3f814a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:00.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 -- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3f815eae0 msgr2=0x7fb3f814a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3f815eae0 0x7fb3f814a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 -- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3f80d2360 msgr2=0x7fb3f81616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3f80d2360 0x7fb3f81616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 -- 172.21.15.12:0/3380907768 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f80cb7f0 con 0x7fb3f815e700 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3fd000640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3f815eae0 0x7fb3f814a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f77fe640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f815e700 0x7fb3f814a090 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec000c00 tx=0x7fb3ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f4ff9640 1 -- 172.21.15.12:0/3380907768 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec035070 con 0x7fb3f815e700 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f4ff9640 1 -- 172.21.15.12:0/3380907768 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3ec004250 con 0x7fb3f815e700 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f80cbad0 con 0x7fb3f815e700 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3f4ff9640 1 -- 172.21.15.12:0/3380907768 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec03a6f0 con 0x7fb3f815e700 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.112+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f80cc090 con 0x7fb3f815e700 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.113+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3f80689e0 con 0x7fb3f815e700 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.114+0000 7fb3f4ff9640 1 -- 172.21.15.12:0/3380907768 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3ec038020 con 0x7fb3f815e700 2024-04-15T15:22:00.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.114+0000 7fb3f4ff9640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:00.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.114+0000 7fb3f4ff9640 1 -- 172.21.15.12:0/3380907768 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fb3ec0adce0 con 0x7fb3f815e700 2024-04-15T15:22:00.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.114+0000 7fb3f7fff640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:00.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.115+0000 7fb3f7fff640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3e406c8f0 0x7fb3e406edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800b9b0 tx=0x7fb3e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.119+0000 7fb3f4ff9640 1 -- 172.21.15.12:0/3380907768 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3ec040070 con 0x7fb3f815e700 2024-04-15T15:22:00.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.376+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7fb3f80ba500 con 0x7fb3f815e700 2024-04-15T15:22:00.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.911+0000 7fb3f4ff9640 1 -- 172.21.15.12:0/3380907768 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v70) ==== 140+0+0 (secure 0 0 0) 0x7fb3ec031800 con 0x7fb3f815e700 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3e406c8f0 msgr2=0x7fb3e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3e406c8f0 0x7fb3e406edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800b9b0 tx=0x7fb3e8008040 comp rx=0 tx=0).stop 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f815e700 msgr2=0x7fb3f814a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f815e700 0x7fb3f814a090 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec000c00 tx=0x7fb3ec0046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 shutdown_connections 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3f815eae0 0x7fb3f814a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f815e700 0x7fb3f814a090 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 --2- 172.21.15.12:0/3380907768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3f80d2360 0x7fb3f81616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.913+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 >> 172.21.15.12:0/3380907768 conn(0x7fb3f808bf60 msgr2=0x7fb3f806e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:00.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.914+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 shutdown_connections 2024-04-15T15:22:00.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:00.914+0000 7fb3fea63640 1 -- 172.21.15.12:0/3380907768 wait complete. 2024-04-15T15:22:00.927 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-15T15:22:01.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.037+0000 7f8d24a7e640 1 Processor -- start 2024-04-15T15:22:01.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.037+0000 7f8d24a7e640 1 -- start start 2024-04-15T15:22:01.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d24a7e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d2015e790 0x7f8d200b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d24a7e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d2015eb70 con 0x7f8d200d1cf0 2024-04-15T15:22:01.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d24a7e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d200d3c80 con 0x7f8d200b7be0 2024-04-15T15:22:01.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d24a7e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8d200d3790 con 0x7f8d2015e790 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d1ed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8d200d1cf0 0x7f8d200d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33614/0 (socket says 172.21.15.12:33614) 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d1ed9d640 1 -- 172.21.15.12:0/1635520361 learned_addr learned my addr 172.21.15.12:0/1635520361 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821688322 0 0) 0x7f8d2015eb70 con 0x7f8d200d1cf0 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d1e59c640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d2015e790 0x7f8d200b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.038+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d0c003520 con 0x7f8d200d1cf0 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2374603629 0 0) 0x7f8d0c003520 con 0x7f8d200d1cf0 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d2015e790 msgr2=0x7f8d200b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d2015e790 0x7f8d200b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 >> v1:172.21.15.55:6789/0 conn(0x7f8d200b7be0 legacy=0x7f8d200d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d200d3f60 con 0x7f8d200d1cf0 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f8d10001690 con 0x7f8d200d1cf0 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 >> v1:172.21.15.12:6789/0 conn(0x7f8d200d1cf0 legacy=0x7f8d200d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d2015e790 0x7f8d0c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d0c003f60 0x7f8d0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d0c004d40 0x7f8d0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8d200d3790 con 0x7f8d0c003f60 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c00bc00 con 0x7f8d0c004d40 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8d0c00be00 con 0x7f8d2015e790 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1dd9b640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d0c003f60 0x7f8d0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1e59c640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d2015e790 0x7f8d0c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1ed9d640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d0c004d40 0x7f8d0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1dd9b640 1 -- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d2015e790 msgr2=0x7f8d0c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1dd9b640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d2015e790 0x7f8d0c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1dd9b640 1 -- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d0c004d40 msgr2=0x7f8d0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1dd9b640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d0c004d40 0x7f8d0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.039+0000 7f8d1dd9b640 1 -- 172.21.15.12:0/1635520361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0c003520 con 0x7f8d0c003f60 2024-04-15T15:22:01.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.040+0000 7f8d1dd9b640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d0c003f60 0x7f8d0c004800 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8d08001750 tx=0x7f8d0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5782f1d4cd9b46ef server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.040+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08001370 con 0x7f8d0c003f60 2024-04-15T15:22:01.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.040+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d080052d0 con 0x7f8d0c003f60 2024-04-15T15:22:01.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.040+0000 7f8d1d59a640 1 -- 172.21.15.12:0/1635520361 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08003830 con 0x7f8d0c003f60 2024-04-15T15:22:01.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.041+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1635520361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d0c003f60 msgr2=0x7f8d0c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.041+0000 7f8d24a7e640 1 --2- 172.21.15.12:0/1635520361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d0c003f60 0x7f8d0c004800 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8d08001750 tx=0x7f8d0c005830 comp rx=0 tx=0).stop 2024-04-15T15:22:01.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.041+0000 7f8d1ed9d640 1 -- 172.21.15.12:0/1635520361 reap_dead start 2024-04-15T15:22:01.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.041+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1635520361 shutdown_connections 2024-04-15T15:22:01.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.041+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1635520361 >> 172.21.15.12:0/1635520361 conn(0x7f8d20067b40 msgr2=0x7f8d20067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:01.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.041+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1635520361 shutdown_connections 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.041+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1635520361 wait complete. 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.042+0000 7f8d24a7e640 1 Processor -- start 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.042+0000 7f8d24a7e640 1 -- start start 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d24a7e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d0c003f60 0x7f8d200caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d24a7e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d20158aa0 0x7f8d20159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d24a7e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d200c79c0 0x7f8d200cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d24a7e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8d200d5150 con 0x7f8d20158aa0 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d24a7e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8d200d45b0 con 0x7f8d200c79c0 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d24a7e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8d200d52d0 con 0x7f8d0c003f60 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d20158aa0 0x7f8d20159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d20158aa0 0x7f8d20159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52922/0 (socket says 172.21.15.12:52922) 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1dd9b640 1 -- 172.21.15.12:0/1661913631 learned_addr learned my addr 172.21.15.12:0/1661913631 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1e59c640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d0c003f60 0x7f8d200caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1ed9d640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d200c79c0 0x7f8d200cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1dd9b640 1 -- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d0c003f60 msgr2=0x7f8d200caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1dd9b640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d0c003f60 0x7f8d200caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1dd9b640 1 -- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d200c79c0 msgr2=0x7f8d200cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1dd9b640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d200c79c0 0x7f8d200cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.043+0000 7f8d1dd9b640 1 -- 172.21.15.12:0/1661913631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d200c7e50 con 0x7f8d20158aa0 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.044+0000 7f8d1dd9b640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d20158aa0 0x7f8d20159fd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005040 tx=0x7f8d08001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.044+0000 7f8d16ffd640 1 -- 172.21.15.12:0/1661913631 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08041020 con 0x7f8d20158aa0 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.044+0000 7f8d16ffd640 1 -- 172.21.15.12:0/1661913631 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d08005480 con 0x7f8d20158aa0 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.044+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d20158f30 con 0x7f8d20158aa0 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.044+0000 7f8d16ffd640 1 -- 172.21.15.12:0/1661913631 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08001370 con 0x7f8d20158aa0 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.044+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d201594f0 con 0x7f8d20158aa0 2024-04-15T15:22:01.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.045+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d200689e0 con 0x7f8d20158aa0 2024-04-15T15:22:01.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.045+0000 7f8d16ffd640 1 -- 172.21.15.12:0/1661913631 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d08044020 con 0x7f8d20158aa0 2024-04-15T15:22:01.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.046+0000 7f8d16ffd640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8d0006c8f0 0x7f8d0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.046+0000 7f8d16ffd640 1 -- 172.21.15.12:0/1661913631 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f8d080ad720 con 0x7f8d20158aa0 2024-04-15T15:22:01.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.046+0000 7f8d1e59c640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8d0006c8f0 0x7f8d0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.047+0000 7f8d1e59c640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8d0006c8f0 0x7f8d0006edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d18014510 tx=0x7f8d1800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.050+0000 7f8d16ffd640 1 -- 172.21.15.12:0/1661913631 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d0804b030 con 0x7f8d20158aa0 2024-04-15T15:22:01.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.235+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8d200b82b0 con 0x7f8d20158aa0 2024-04-15T15:22:01.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.238+0000 7f8d16ffd640 1 -- 172.21.15.12:0/1661913631 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f8d0807b0c0 con 0x7f8d20158aa0 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8d0006c8f0 msgr2=0x7f8d0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8d0006c8f0 0x7f8d0006edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d18014510 tx=0x7f8d1800b040 comp rx=0 tx=0).stop 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d20158aa0 msgr2=0x7f8d20159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d20158aa0 0x7f8d20159fd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005040 tx=0x7f8d08001120 comp rx=0 tx=0).stop 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 shutdown_connections 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8d0006c8f0 0x7f8d0006edb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d200c79c0 0x7f8d200cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d20158aa0 0x7f8d20159fd0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 --2- 172.21.15.12:0/1661913631 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8d0c003f60 0x7f8d200caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 >> 172.21.15.12:0/1661913631 conn(0x7f8d20067b40 msgr2=0x7f8d200b8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 shutdown_connections 2024-04-15T15:22:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.240+0000 7f8d24a7e640 1 -- 172.21.15.12:0/1661913631 wait complete. 2024-04-15T15:22:01.253 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-15T15:22:01.253 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:22:01.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:22:01.357 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-15T15:22:01.357 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:22:01.357 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:01.373 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:22:01.373 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:01.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:01.472 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:22:01.524 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:22:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.576+0000 7f3e5fa75640 1 Processor -- start 2024-04-15T15:22:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.576+0000 7f3e5fa75640 1 -- start start 2024-04-15T15:22:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5fa75640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e5815ecb0 0x7f3e580b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5fa75640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3e580b1d30 0x7f3e580af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5fa75640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e580b10d0 con 0x7f3e580b4350 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5fa75640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3e5815f090 con 0x7f3e580b1d30 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5fa75640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3e580cdf80 con 0x7f3e5815ecb0 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5e012640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3e580b1d30 0x7f3e580af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5d010640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3e580b4350 0x7f3e580b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33616/0 (socket says 172.21.15.12:33616) 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5d010640 1 -- 172.21.15.12:0/3425168467 learned_addr learned my addr 172.21.15.12:0/3425168467 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5d811640 1 --2- 172.21.15.12:0/3425168467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e5815ecb0 0x7f3e580b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5e012640 1 -- 172.21.15.12:0/3425168467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e5815ecb0 msgr2=0x7f3e580b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5e012640 1 --2- 172.21.15.12:0/3425168467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e5815ecb0 0x7f3e580b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5e012640 1 -- 172.21.15.12:0/3425168467 >> v1:172.21.15.12:6789/0 conn(0x7f3e580b4350 legacy=0x7f3e580b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5e012640 1 -- 172.21.15.12:0/3425168467 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e580af920 con 0x7f3e580b1d30 2024-04-15T15:22:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.577+0000 7f3e5d811640 1 --2- 172.21.15.12:0/3425168467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e5815ecb0 0x7f3e580b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.578+0000 7f3e5e012640 1 --2- 172.21.15.12:0/3425168467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b1d30 0x7f3e580af340 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009d90 tx=0x7f3e5002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86bd40e44aebc3a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.578+0000 7f3e5c80f640 1 -- 172.21.15.12:0/3425168467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e50039070 con 0x7f3e580b1d30 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.578+0000 7f3e5c80f640 1 -- 172.21.15.12:0/3425168467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e5002cce0 con 0x7f3e580b1d30 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.578+0000 7f3e5c80f640 1 -- 172.21.15.12:0/3425168467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e50034660 con 0x7f3e580b1d30 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.579+0000 7f3e5fa75640 1 -- 172.21.15.12:0/3425168467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b1d30 msgr2=0x7f3e580af340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.579+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/3425168467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b1d30 0x7f3e580af340 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009d90 tx=0x7f3e5002bf30 comp rx=0 tx=0).stop 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.579+0000 7f3e5fa75640 1 -- 172.21.15.12:0/3425168467 shutdown_connections 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.579+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/3425168467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b1d30 0x7f3e580af340 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.579+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/3425168467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e5815ecb0 0x7f3e580b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.579+0000 7f3e5fa75640 1 -- 172.21.15.12:0/3425168467 >> 172.21.15.12:0/3425168467 conn(0x7f3e580b6870 msgr2=0x7f3e580b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.579+0000 7f3e5fa75640 1 -- 172.21.15.12:0/3425168467 shutdown_connections 2024-04-15T15:22:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.579+0000 7f3e5fa75640 1 -- 172.21.15.12:0/3425168467 wait complete. 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.580+0000 7f3e5fa75640 1 Processor -- start 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.580+0000 7f3e5fa75640 1 -- start start 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5fa75640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e580b1d30 0x7f3e581f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5fa75640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b4350 0x7f3e581f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5fa75640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5815ecb0 0x7f3e580cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5fa75640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3e580b4790 con 0x7f3e580b4350 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5fa75640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3e580afee0 con 0x7f3e5815ecb0 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5fa75640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3e580b4910 con 0x7f3e580b1d30 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b4350 0x7f3e581f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b4350 0x7f3e581f5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52934/0 (socket says 172.21.15.12:52934) 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 -- 172.21.15.12:0/1649577741 learned_addr learned my addr 172.21.15.12:0/1649577741 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d811640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e580b1d30 0x7f3e581f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5e012640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5815ecb0 0x7f3e580cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 -- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e580b1d30 msgr2=0x7f3e581f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e580b1d30 0x7f3e581f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 -- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5815ecb0 msgr2=0x7f3e580cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5815ecb0 0x7f3e580cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 -- 172.21.15.12:0/1649577741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e581f8160 con 0x7f3e580b4350 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.581+0000 7f3e5d010640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b4350 0x7f3e581f5900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800c400 tx=0x7f3e4800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.582+0000 7f3e4e7fc640 1 -- 172.21.15.12:0/1649577741 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48017070 con 0x7f3e580b4350 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.582+0000 7f3e4e7fc640 1 -- 172.21.15.12:0/1649577741 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e48005ce0 con 0x7f3e580b4350 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.582+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5815f3b0 con 0x7f3e580b4350 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.582+0000 7f3e4e7fc640 1 -- 172.21.15.12:0/1649577741 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e480072f0 con 0x7f3e580b4350 2024-04-15T15:22:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.582+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5815f970 con 0x7f3e580b4350 2024-04-15T15:22:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.583+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e580689e0 con 0x7f3e580b4350 2024-04-15T15:22:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.583+0000 7f3e4e7fc640 1 -- 172.21.15.12:0/1649577741 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e4801a070 con 0x7f3e580b4350 2024-04-15T15:22:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.583+0000 7f3e4e7fc640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e3c06c8f0 0x7f3e3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.584+0000 7f3e4e7fc640 1 -- 172.21.15.12:0/1649577741 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f3e4808b2c0 con 0x7f3e580b4350 2024-04-15T15:22:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.584+0000 7f3e5d811640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e3c06c8f0 0x7f3e3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.584+0000 7f3e5d811640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e3c06c8f0 0x7f3e3c06edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400aac0 tx=0x7f3e54008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.588+0000 7f3e4e7fc640 1 -- 172.21.15.12:0/1649577741 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e581f8160 con 0x7f3e580b4350 2024-04-15T15:22:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.782+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e5806c7c0 con 0x7f3e580b4350 2024-04-15T15:22:01.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.783+0000 7f3e4e7fc640 1 -- 172.21.15.12:0/1649577741 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f3e4808b0e0 con 0x7f3e580b4350 2024-04-15T15:22:01.785 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:01.785 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:01.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e3c06c8f0 msgr2=0x7f3e3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e3c06c8f0 0x7f3e3c06edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400aac0 tx=0x7f3e54008040 comp rx=0 tx=0).stop 2024-04-15T15:22:01.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b4350 msgr2=0x7f3e581f5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b4350 0x7f3e581f5900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800c400 tx=0x7f3e4800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 shutdown_connections 2024-04-15T15:22:01.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e3c06c8f0 0x7f3e3c06edb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5815ecb0 0x7f3e580cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e580b4350 0x7f3e581f5900 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 --2- 172.21.15.12:0/1649577741 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e580b1d30 0x7f3e581f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.785+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 >> 172.21.15.12:0/1649577741 conn(0x7f3e580b6870 msgr2=0x7f3e58074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:01.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.786+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 shutdown_connections 2024-04-15T15:22:01.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.786+0000 7f3e5fa75640 1 -- 172.21.15.12:0/1649577741 wait complete. 2024-04-15T15:22:01.798 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:01.799 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:01.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.906+0000 7fb672fa8640 1 Processor -- start 2024-04-15T15:22:01.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.907+0000 7fb672fa8640 1 -- start start 2024-04-15T15:22:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.907+0000 7fb672fa8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c15e700 0x7fb66c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.907+0000 7fb672fa8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb66c0d2360 0x7fb66c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.907+0000 7fb672fa8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb66c15eae0 0x7fb66c0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.907+0000 7fb672fa8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb66c0b53b0 con 0x7fb66c15eae0 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.907+0000 7fb672fa8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb66c0b47a0 con 0x7fb66c0d2360 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.907+0000 7fb672fa8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb66c0b4a80 con 0x7fb66c15e700 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.907+0000 7fb671545640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb66c15eae0 0x7fb66c0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb671545640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb66c15eae0 0x7fb66c0d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52952/0 (socket says 172.21.15.12:52952) 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb671545640 1 -- 172.21.15.12:0/247235514 learned_addr learned my addr 172.21.15.12:0/247235514 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb670d44640 1 --2- 172.21.15.12:0/247235514 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c15e700 0x7fb66c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb66bfff640 1 --2- 172.21.15.12:0/247235514 >> v2:172.21.15.55:3300/0 conn(0x7fb66c0d2360 0x7fb66c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb671545640 1 -- 172.21.15.12:0/247235514 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c15e700 msgr2=0x7fb66c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb671545640 1 --2- 172.21.15.12:0/247235514 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c15e700 0x7fb66c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb671545640 1 -- 172.21.15.12:0/247235514 >> v2:172.21.15.55:3300/0 conn(0x7fb66c0d2360 msgr2=0x7fb66c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb671545640 1 --2- 172.21.15.12:0/247235514 >> v2:172.21.15.55:3300/0 conn(0x7fb66c0d2360 0x7fb66c0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb671545640 1 -- 172.21.15.12:0/247235514 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c0b4d60 con 0x7fb66c15eae0 2024-04-15T15:22:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.908+0000 7fb66bfff640 1 --2- 172.21.15.12:0/247235514 >> v2:172.21.15.55:3300/0 conn(0x7fb66c0d2360 0x7fb66c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.909+0000 7fb671545640 1 --2- 172.21.15.12:0/247235514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15eae0 0x7fb66c0d30a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0d2100 tx=0x7fb66402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d550c46a7ddeb195 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.909+0000 7fb66b7fe640 1 -- 172.21.15.12:0/247235514 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66402ca70 con 0x7fb66c15eae0 2024-04-15T15:22:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.909+0000 7fb66b7fe640 1 -- 172.21.15.12:0/247235514 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb66402cc10 con 0x7fb66c15eae0 2024-04-15T15:22:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.909+0000 7fb66b7fe640 1 -- 172.21.15.12:0/247235514 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6640343f0 con 0x7fb66c15eae0 2024-04-15T15:22:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 -- 172.21.15.12:0/247235514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15eae0 msgr2=0x7fb66c0d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 --2- 172.21.15.12:0/247235514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15eae0 0x7fb66c0d30a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0d2100 tx=0x7fb66402bed0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 -- 172.21.15.12:0/247235514 shutdown_connections 2024-04-15T15:22:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 --2- 172.21.15.12:0/247235514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15eae0 0x7fb66c0d30a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 --2- 172.21.15.12:0/247235514 >> v2:172.21.15.55:3300/0 conn(0x7fb66c0d2360 0x7fb66c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 --2- 172.21.15.12:0/247235514 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c15e700 0x7fb66c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 -- 172.21.15.12:0/247235514 >> 172.21.15.12:0/247235514 conn(0x7fb66c08bf60 msgr2=0x7fb66c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 -- 172.21.15.12:0/247235514 shutdown_connections 2024-04-15T15:22:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.910+0000 7fb672fa8640 1 -- 172.21.15.12:0/247235514 wait complete. 2024-04-15T15:22:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.911+0000 7fb672fa8640 1 Processor -- start 2024-04-15T15:22:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.911+0000 7fb672fa8640 1 -- start start 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb672fa8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c0d2360 0x7fb66c152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb672fa8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15e700 0x7fb66c15f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb672fa8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb66c15eae0 0x7fb66c0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb672fa8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c0b5d80 con 0x7fb66c15e700 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb672fa8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c0b4450 con 0x7fb66c15eae0 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb672fa8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb66c0d1d40 con 0x7fb66c0d2360 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb66bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15e700 0x7fb66c15f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb66bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15e700 0x7fb66c15f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52954/0 (socket says 172.21.15.12:52954) 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb66bfff640 1 -- 172.21.15.12:0/839999959 learned_addr learned my addr 172.21.15.12:0/839999959 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb670d44640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c0d2360 0x7fb66c152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb671545640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb66c15eae0 0x7fb66c0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb66bfff640 1 -- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c0d2360 msgr2=0x7fb66c152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb66bfff640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c0d2360 0x7fb66c152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb66bfff640 1 -- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb66c15eae0 msgr2=0x7fb66c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb66bfff640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb66c15eae0 0x7fb66c0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb66bfff640 1 -- 172.21.15.12:0/839999959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c1603f0 con 0x7fb66c15e700 2024-04-15T15:22:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.912+0000 7fb671545640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb66c15eae0 0x7fb66c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.913+0000 7fb66bfff640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15e700 0x7fb66c15f740 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb65800d830 tx=0x7fb65800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.914+0000 7fb6697fa640 1 -- 172.21.15.12:0/839999959 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658018070 con 0x7fb66c15e700 2024-04-15T15:22:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.914+0000 7fb6697fa640 1 -- 172.21.15.12:0/839999959 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65800be10 con 0x7fb66c15e700 2024-04-15T15:22:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.914+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66c1606d0 con 0x7fb66c15e700 2024-04-15T15:22:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.915+0000 7fb6697fa640 1 -- 172.21.15.12:0/839999959 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658004250 con 0x7fb66c15e700 2024-04-15T15:22:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.915+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66c15f220 con 0x7fb66c15e700 2024-04-15T15:22:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.916+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb66c160bf0 con 0x7fb66c15e700 2024-04-15T15:22:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.916+0000 7fb6697fa640 1 -- 172.21.15.12:0/839999959 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb658009850 con 0x7fb66c15e700 2024-04-15T15:22:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.917+0000 7fb6697fa640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb65006c8f0 0x7fb65006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.917+0000 7fb6697fa640 1 -- 172.21.15.12:0/839999959 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fb65808c1f0 con 0x7fb66c15e700 2024-04-15T15:22:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.917+0000 7fb670d44640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb65006c8f0 0x7fb65006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.918+0000 7fb670d44640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb65006c8f0 0x7fb65006edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb660009ac0 tx=0x7fb660008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:01.925+0000 7fb6697fa640 1 -- 172.21.15.12:0/839999959 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb65805a1e0 con 0x7fb66c15e700 2024-04-15T15:22:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.110+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb66c0689e0 con 0x7fb66c15e700 2024-04-15T15:22:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.110+0000 7fb6697fa640 1 -- 172.21.15.12:0/839999959 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fb658059ae0 con 0x7fb66c15e700 2024-04-15T15:22:02.112 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb65006c8f0 msgr2=0x7fb65006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb65006c8f0 0x7fb65006edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb660009ac0 tx=0x7fb660008040 comp rx=0 tx=0).stop 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15e700 msgr2=0x7fb66c15f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15e700 0x7fb66c15f740 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb65800d830 tx=0x7fb65800dcf0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 shutdown_connections 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb65006c8f0 0x7fb65006edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb66c15eae0 0x7fb66c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb66c15e700 0x7fb66c15f740 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 --2- 172.21.15.12:0/839999959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb66c0d2360 0x7fb66c152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 >> 172.21.15.12:0/839999959 conn(0x7fb66c08bf60 msgr2=0x7fb66c0cfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 shutdown_connections 2024-04-15T15:22:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.112+0000 7fb672fa8640 1 -- 172.21.15.12:0/839999959 wait complete. 2024-04-15T15:22:02.125 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.231+0000 7fc1cc5f4640 1 Processor -- start 2024-04-15T15:22:02.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.231+0000 7fc1cc5f4640 1 -- start start 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1cc5f4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc1c4072d80 0x7fc1c4071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1cc5f4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1c40af3e0 0x7fc1c40af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1cc5f4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1c415db40 con 0x7fc1c40d0690 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1cc5f4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c415cfc0 con 0x7fc1c40af3e0 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1cc5f4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c415d2a0 con 0x7fc1c4072d80 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1c9b8f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1c40af3e0 0x7fc1c40af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1c9b8f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1c40af3e0 0x7fc1c40af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52700/0 (socket says 172.21.15.12:52700) 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1cab91640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc1c40d0690 0x7fc1c406e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34648/0 (socket says 172.21.15.12:34648) 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1c9b8f640 1 -- 172.21.15.12:0/1966302405 learned_addr learned my addr 172.21.15.12:0/1966302405 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1ca390640 1 --2- 172.21.15.12:0/1966302405 >> v2:172.21.15.55:3300/0 conn(0x7fc1c4072d80 0x7fc1c4071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1c9b8f640 1 -- 172.21.15.12:0/1966302405 >> v2:172.21.15.55:3300/0 conn(0x7fc1c4072d80 msgr2=0x7fc1c4071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1c9b8f640 1 --2- 172.21.15.12:0/1966302405 >> v2:172.21.15.55:3300/0 conn(0x7fc1c4072d80 0x7fc1c4071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1c9b8f640 1 -- 172.21.15.12:0/1966302405 >> v1:172.21.15.12:6789/0 conn(0x7fc1c40d0690 legacy=0x7fc1c406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1c9b8f640 1 -- 172.21.15.12:0/1966302405 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c415d580 con 0x7fc1c40af3e0 2024-04-15T15:22:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.232+0000 7fc1c938e640 1 -- 172.21.15.12:0/1966302405 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152641788 0 0) 0x7fc1c415db40 con 0x7fc1c40d0690 2024-04-15T15:22:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.233+0000 7fc1c9b8f640 1 --2- 172.21.15.12:0/1966302405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c40af3e0 0x7fc1c40af840 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40d11c0 tx=0x7fc1b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed76cfc470b69d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.233+0000 7fc1c938e640 1 -- 172.21.15.12:0/1966302405 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8035020 con 0x7fc1c40af3e0 2024-04-15T15:22:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.233+0000 7fc1c938e640 1 -- 172.21.15.12:0/1966302405 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1b8031870 con 0x7fc1c40af3e0 2024-04-15T15:22:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.233+0000 7fc1c938e640 1 -- 172.21.15.12:0/1966302405 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8031c40 con 0x7fc1c40af3e0 2024-04-15T15:22:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.234+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/1966302405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c40af3e0 msgr2=0x7fc1c40af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.234+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/1966302405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c40af3e0 0x7fc1c40af840 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40d11c0 tx=0x7fc1b802b910 comp rx=0 tx=0).stop 2024-04-15T15:22:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.234+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/1966302405 shutdown_connections 2024-04-15T15:22:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.234+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/1966302405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c40af3e0 0x7fc1c40af840 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.234+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/1966302405 >> v2:172.21.15.55:3300/0 conn(0x7fc1c4072d80 0x7fc1c4071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.234+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/1966302405 >> 172.21.15.12:0/1966302405 conn(0x7fc1c406be00 msgr2=0x7fc1c406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.234+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/1966302405 shutdown_connections 2024-04-15T15:22:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.234+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/1966302405 wait complete. 2024-04-15T15:22:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.235+0000 7fc1cc5f4640 1 Processor -- start 2024-04-15T15:22:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.235+0000 7fc1cc5f4640 1 -- start start 2024-04-15T15:22:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1cc5f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c4072d80 0x7fc1c4150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1cc5f4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1c40af3e0 0x7fc1c40ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1cc5f4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1c40d0690 0x7fc1c41549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1cc5f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c415cd10 con 0x7fc1c4072d80 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1cc5f4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c415cb90 con 0x7fc1c40af3e0 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1cc5f4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c40d0b00 con 0x7fc1c40d0690 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1ca390640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c4072d80 0x7fc1c4150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1ca390640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c4072d80 0x7fc1c4150680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52704/0 (socket says 172.21.15.12:52704) 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1ca390640 1 -- 172.21.15.12:0/4142096270 learned_addr learned my addr 172.21.15.12:0/4142096270 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1cab91640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1c40d0690 0x7fc1c41549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1c9b8f640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1c40af3e0 0x7fc1c40ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1ca390640 1 -- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1c40d0690 msgr2=0x7fc1c41549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1ca390640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1c40d0690 0x7fc1c41549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1ca390640 1 -- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1c40af3e0 msgr2=0x7fc1c40ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1ca390640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1c40af3e0 0x7fc1c40ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1ca390640 1 -- 172.21.15.12:0/4142096270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c40ccd20 con 0x7fc1c4072d80 2024-04-15T15:22:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1cab91640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1c40d0690 0x7fc1c41549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.236+0000 7fc1c9b8f640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1c40af3e0 0x7fc1c40ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.237+0000 7fc1ca390640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c4072d80 0x7fc1c4150680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000c7f0 tx=0x7fc1c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.237+0000 7fc1b6ffd640 1 -- 172.21.15.12:0/4142096270 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0015070 con 0x7fc1c4072d80 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.237+0000 7fc1b6ffd640 1 -- 172.21.15.12:0/4142096270 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1c0004240 con 0x7fc1c4072d80 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.237+0000 7fc1b6ffd640 1 -- 172.21.15.12:0/4142096270 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0007620 con 0x7fc1c4072d80 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.237+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c40cd000 con 0x7fc1c4072d80 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.237+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c40cd5c0 con 0x7fc1c4072d80 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.238+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c40689e0 con 0x7fc1c4072d80 2024-04-15T15:22:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.238+0000 7fc1b6ffd640 1 -- 172.21.15.12:0/4142096270 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc1c00079d0 con 0x7fc1c4072d80 2024-04-15T15:22:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.239+0000 7fc1b6ffd640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1a806c8f0 0x7fc1a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.239+0000 7fc1b6ffd640 1 -- 172.21.15.12:0/4142096270 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fc1c008c420 con 0x7fc1c4072d80 2024-04-15T15:22:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.239+0000 7fc1c9b8f640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1a806c8f0 0x7fc1a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.240+0000 7fc1c9b8f640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1a806c8f0 0x7fc1a806edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8036000 tx=0x7fc1b803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.243+0000 7fc1b6ffd640 1 -- 172.21.15.12:0/4142096270 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1c005a410 con 0x7fc1c4072d80 2024-04-15T15:22:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.428+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc1c406cd10 con 0x7fc1c4072d80 2024-04-15T15:22:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.428+0000 7fc1b6ffd640 1 -- 172.21.15.12:0/4142096270 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fc1c0059d10 con 0x7fc1c4072d80 2024-04-15T15:22:02.430 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.430+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1a806c8f0 msgr2=0x7fc1a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.430+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1a806c8f0 0x7fc1a806edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8036000 tx=0x7fc1b803c040 comp rx=0 tx=0).stop 2024-04-15T15:22:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.430+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c4072d80 msgr2=0x7fc1c4150680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.430+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c4072d80 0x7fc1c4150680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000c7f0 tx=0x7fc1c000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.431+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 shutdown_connections 2024-04-15T15:22:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.431+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc1c40d0690 0x7fc1c41549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.431+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc1a806c8f0 0x7fc1a806edb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.431+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1c40af3e0 0x7fc1c40ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.431+0000 7fc1cc5f4640 1 --2- 172.21.15.12:0/4142096270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c4072d80 0x7fc1c4150680 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.431+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 >> 172.21.15.12:0/4142096270 conn(0x7fc1c406be00 msgr2=0x7fc1c41566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.431+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 shutdown_connections 2024-04-15T15:22:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.431+0000 7fc1cc5f4640 1 -- 172.21.15.12:0/4142096270 wait complete. 2024-04-15T15:22:02.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.549+0000 7f78c47a8640 1 Processor -- start 2024-04-15T15:22:02.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.550+0000 7f78c47a8640 1 -- start start 2024-04-15T15:22:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.550+0000 7f78c47a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d3240 0x7f78bc0d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.550+0000 7f78c47a8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f78bc0d1440 0x7f78bc0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.550+0000 7f78c47a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78bc0d2480 con 0x7f78bc15e790 2024-04-15T15:22:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.550+0000 7f78c47a8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f78bc0d2db0 con 0x7f78bc0d1440 2024-04-15T15:22:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.550+0000 7f78c47a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f78bc0d2760 con 0x7f78bc0d3240 2024-04-15T15:22:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c1d43640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d3240 0x7f78bc0d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c2d45640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f78bc0d1440 0x7f78bc0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c1d43640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d3240 0x7f78bc0d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:39524/0 (socket says 172.21.15.12:39524) 2024-04-15T15:22:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c2d45640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f78bc0d1440 0x7f78bc0b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:42626/0 (socket says 172.21.15.12:42626) 2024-04-15T15:22:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c2544640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f78bc15e790 0x7f78bc0cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:60250/0 (socket says 172.21.15.12:60250) 2024-04-15T15:22:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c1d43640 1 -- 172.21.15.12:0/3207744151 learned_addr learned my addr 172.21.15.12:0/3207744151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c2d45640 1 -- 172.21.15.12:0/3207744151 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d3240 msgr2=0x7f78bc0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c2d45640 1 --2- 172.21.15.12:0/3207744151 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d3240 0x7f78bc0d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c2d45640 1 -- 172.21.15.12:0/3207744151 >> v1:172.21.15.55:6789/0 conn(0x7f78bc15e790 legacy=0x7f78bc0cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c2d45640 1 -- 172.21.15.12:0/3207744151 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc0d2a40 con 0x7f78bc0d1440 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.551+0000 7f78c1d43640 1 --2- 172.21.15.12:0/3207744151 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d3240 0x7f78bc0d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.552+0000 7f78c2d45640 1 --2- 172.21.15.12:0/3207744151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc0d1440 0x7f78bc0b74f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009d90 tx=0x7f78b402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=884b9d46175bf51 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.552+0000 7f78c1542640 1 -- 172.21.15.12:0/3207744151 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4039070 con 0x7f78bc0d1440 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.552+0000 7f78c1542640 1 -- 172.21.15.12:0/3207744151 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b402cc50 con 0x7f78bc0d1440 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.552+0000 7f78c1542640 1 -- 172.21.15.12:0/3207744151 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4034440 con 0x7f78bc0d1440 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.553+0000 7f78c47a8640 1 -- 172.21.15.12:0/3207744151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc0d1440 msgr2=0x7f78bc0b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.553+0000 7f78c47a8640 1 --2- 172.21.15.12:0/3207744151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc0d1440 0x7f78bc0b74f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009d90 tx=0x7f78b402bea0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.553+0000 7f78c47a8640 1 -- 172.21.15.12:0/3207744151 shutdown_connections 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.553+0000 7f78c47a8640 1 --2- 172.21.15.12:0/3207744151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc0d1440 0x7f78bc0b74f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.553+0000 7f78c47a8640 1 --2- 172.21.15.12:0/3207744151 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d3240 0x7f78bc0d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.553+0000 7f78c47a8640 1 -- 172.21.15.12:0/3207744151 >> 172.21.15.12:0/3207744151 conn(0x7f78bc08bf60 msgr2=0x7f78bc0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.553+0000 7f78c47a8640 1 -- 172.21.15.12:0/3207744151 shutdown_connections 2024-04-15T15:22:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.553+0000 7f78c47a8640 1 -- 172.21.15.12:0/3207744151 wait complete. 2024-04-15T15:22:02.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.554+0000 7f78c47a8640 1 Processor -- start 2024-04-15T15:22:02.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.554+0000 7f78c47a8640 1 -- start start 2024-04-15T15:22:02.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c47a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d1440 0x7f78bc0cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c47a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78bc0d3240 0x7f78bc149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c47a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc15e790 0x7f78bc0cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c47a8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc0d5180 con 0x7f78bc0d3240 2024-04-15T15:22:02.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c47a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc0d0430 con 0x7f78bc15e790 2024-04-15T15:22:02.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c47a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f78bc0d5300 con 0x7f78bc0d1440 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78bc0d3240 0x7f78bc149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78bc0d3240 0x7f78bc149a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52708/0 (socket says 172.21.15.12:52708) 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 -- 172.21.15.12:0/417842066 learned_addr learned my addr 172.21.15.12:0/417842066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c2544640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d1440 0x7f78bc0cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c2d45640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc15e790 0x7f78bc0cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 -- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d1440 msgr2=0x7f78bc0cb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d1440 0x7f78bc0cb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 -- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc15e790 msgr2=0x7f78bc0cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc15e790 0x7f78bc0cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 -- 172.21.15.12:0/417842066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc156520 con 0x7f78bc0d3240 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.555+0000 7f78c1d43640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78bc0d3240 0x7f78bc149a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f78b800e9d0 tx=0x7f78b800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.556+0000 7f78aeffd640 1 -- 172.21.15.12:0/417842066 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8017070 con 0x7f78bc0d3240 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.556+0000 7f78aeffd640 1 -- 172.21.15.12:0/417842066 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b800ca60 con 0x7f78bc0d3240 2024-04-15T15:22:02.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.556+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78bc0cbe70 con 0x7f78bc0d3240 2024-04-15T15:22:02.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.556+0000 7f78aeffd640 1 -- 172.21.15.12:0/417842066 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b800ce10 con 0x7f78bc0d3240 2024-04-15T15:22:02.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.556+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78bc0cc430 con 0x7f78bc0d3240 2024-04-15T15:22:02.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.557+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78bc0689e0 con 0x7f78bc0d3240 2024-04-15T15:22:02.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.557+0000 7f78aeffd640 1 -- 172.21.15.12:0/417842066 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f78b801a070 con 0x7f78bc0d3240 2024-04-15T15:22:02.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.558+0000 7f78aeffd640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78a006c920 0x7f78a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.558+0000 7f78aeffd640 1 -- 172.21.15.12:0/417842066 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f78b808be40 con 0x7f78bc0d3240 2024-04-15T15:22:02.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.558+0000 7f78c2544640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78a006c920 0x7f78a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.558+0000 7f78c2544640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78a006c920 0x7f78a006ede0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f78b0007fd0 tx=0x7f78b0007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.562+0000 7f78aeffd640 1 -- 172.21.15.12:0/417842066 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78bc156520 con 0x7f78bc0d3240 2024-04-15T15:22:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.747+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78bc0cea60 con 0x7f78bc0d3240 2024-04-15T15:22:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.748+0000 7f78aeffd640 1 -- 172.21.15.12:0/417842066 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f78b80597e0 con 0x7f78bc0d3240 2024-04-15T15:22:02.749 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78a006c920 msgr2=0x7f78a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78a006c920 0x7f78a006ede0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f78b0007fd0 tx=0x7f78b0007e80 comp rx=0 tx=0).stop 2024-04-15T15:22:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78bc0d3240 msgr2=0x7f78bc149a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78bc0d3240 0x7f78bc149a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f78b800e9d0 tx=0x7f78b800ee90 comp rx=0 tx=0).stop 2024-04-15T15:22:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 shutdown_connections 2024-04-15T15:22:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78a006c920 0x7f78a006ede0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78bc15e790 0x7f78bc0cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78bc0d3240 0x7f78bc149a90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 --2- 172.21.15.12:0/417842066 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78bc0d1440 0x7f78bc0cb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 >> 172.21.15.12:0/417842066 conn(0x7f78bc08bf60 msgr2=0x7f78bc06cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 shutdown_connections 2024-04-15T15:22:02.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.750+0000 7f78c47a8640 1 -- 172.21.15.12:0/417842066 wait complete. 2024-04-15T15:22:02.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.868+0000 7fe6eaff2640 1 Processor -- start 2024-04-15T15:22:02.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.868+0000 7fe6eaff2640 1 -- start start 2024-04-15T15:22:02.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.869+0000 7fe6eaff2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e415c8d0 0x7fe6e40cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.869+0000 7fe6eaff2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6e409b650 0x7fe6e409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.869+0000 7fe6eaff2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6e40d0350 con 0x7fe6e409af10 2024-04-15T15:22:02.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.869+0000 7fe6eaff2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe6e40b5b00 con 0x7fe6e409b650 2024-04-15T15:22:02.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.869+0000 7fe6eaff2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e40ce100 con 0x7fe6e415c8d0 2024-04-15T15:22:02.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.869+0000 7fe6e3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6e409b650 0x7fe6e409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.869+0000 7fe6e958f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe6e409af10 0x7fe6e40d0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34660/0 (socket says 172.21.15.12:34660) 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.869+0000 7fe6e958f640 1 -- 172.21.15.12:0/557515629 learned_addr learned my addr 172.21.15.12:0/557515629 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.870+0000 7fe6e8d8e640 1 --2- 172.21.15.12:0/557515629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e415c8d0 0x7fe6e40cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.870+0000 7fe6e3fff640 1 -- 172.21.15.12:0/557515629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e415c8d0 msgr2=0x7fe6e40cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.870+0000 7fe6e37fe640 1 -- 172.21.15.12:0/557515629 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4051729726 0 0) 0x7fe6e40d0350 con 0x7fe6e409af10 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.870+0000 7fe6e3fff640 1 --2- 172.21.15.12:0/557515629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e415c8d0 0x7fe6e40cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.870+0000 7fe6e3fff640 1 -- 172.21.15.12:0/557515629 >> v1:172.21.15.12:6789/0 conn(0x7fe6e409af10 legacy=0x7fe6e40d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.870+0000 7fe6e3fff640 1 -- 172.21.15.12:0/557515629 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e40b54b0 con 0x7fe6e409b650 2024-04-15T15:22:02.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.870+0000 7fe6e8d8e640 1 --2- 172.21.15.12:0/557515629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e415c8d0 0x7fe6e40cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.870+0000 7fe6e3fff640 1 --2- 172.21.15.12:0/557515629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409b650 0x7fe6e409a9d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80097a0 tx=0x7fe6d802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1a96bb90d547fe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6e37fe640 1 -- 172.21.15.12:0/557515629 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80315a0 con 0x7fe6e409b650 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6e37fe640 1 -- 172.21.15.12:0/557515629 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d8037070 con 0x7fe6e409b650 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6e37fe640 1 -- 172.21.15.12:0/557515629 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8031960 con 0x7fe6e409b650 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6eaff2640 1 -- 172.21.15.12:0/557515629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409b650 msgr2=0x7fe6e409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/557515629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409b650 0x7fe6e409a9d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80097a0 tx=0x7fe6d802b840 comp rx=0 tx=0).stop 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6eaff2640 1 -- 172.21.15.12:0/557515629 shutdown_connections 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/557515629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409b650 0x7fe6e409a9d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/557515629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e415c8d0 0x7fe6e40cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6eaff2640 1 -- 172.21.15.12:0/557515629 >> 172.21.15.12:0/557515629 conn(0x7fe6e40b3fb0 msgr2=0x7fe6e40b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6eaff2640 1 -- 172.21.15.12:0/557515629 shutdown_connections 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.871+0000 7fe6eaff2640 1 -- 172.21.15.12:0/557515629 wait complete. 2024-04-15T15:22:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.872+0000 7fe6eaff2640 1 Processor -- start 2024-04-15T15:22:02.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.872+0000 7fe6eaff2640 1 -- start start 2024-04-15T15:22:02.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6eaff2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409af10 0x7fe6e4158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6eaff2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e409b650 0x7fe6e406d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6eaff2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6e406dd40 0x7fe6e40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6eaff2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e40b7120 con 0x7fe6e409af10 2024-04-15T15:22:02.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6eaff2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e40b6fa0 con 0x7fe6e406dd40 2024-04-15T15:22:02.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6eaff2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e40b72a0 con 0x7fe6e409b650 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6e8d8e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409af10 0x7fe6e4158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6e8d8e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409af10 0x7fe6e4158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52730/0 (socket says 172.21.15.12:52730) 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.873+0000 7fe6e8d8e640 1 -- 172.21.15.12:0/3557183020 learned_addr learned my addr 172.21.15.12:0/3557183020 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e3fff640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e409b650 0x7fe6e406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e958f640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6e406dd40 0x7fe6e40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e8d8e640 1 -- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e409b650 msgr2=0x7fe6e406d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e8d8e640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e409b650 0x7fe6e406d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e8d8e640 1 -- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6e406dd40 msgr2=0x7fe6e40710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e8d8e640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6e406dd40 0x7fe6e40710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e8d8e640 1 -- 172.21.15.12:0/3557183020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e41f7960 con 0x7fe6e409af10 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e3fff640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e409b650 0x7fe6e406d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e958f640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6e406dd40 0x7fe6e40710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:02.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e8d8e640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409af10 0x7fe6e4158b50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00cf60 tx=0x7fe6dc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e17fa640 1 -- 172.21.15.12:0/3557183020 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc01a070 con 0x7fe6e409af10 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e17fa640 1 -- 172.21.15.12:0/3557183020 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6dc00d840 con 0x7fe6e409af10 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e41f7c00 con 0x7fe6e409af10 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6e17fa640 1 -- 172.21.15.12:0/3557183020 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc00dc10 con 0x7fe6e409af10 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.874+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e41f81c0 con 0x7fe6e409af10 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.875+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6e40690f0 con 0x7fe6e409af10 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.875+0000 7fe6e17fa640 1 -- 172.21.15.12:0/3557183020 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe6dc01d070 con 0x7fe6e409af10 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.876+0000 7fe6e17fa640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c920 0x7fe6d006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.876+0000 7fe6e17fa640 1 -- 172.21.15.12:0/3557183020 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fe6dc08c770 con 0x7fe6e409af10 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.876+0000 7fe6e3fff640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c920 0x7fe6d006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.877+0000 7fe6e3fff640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c920 0x7fe6d006ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8036000 tx=0x7fe6d8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:02.880+0000 7fe6e17fa640 1 -- 172.21.15.12:0/3557183020 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6dc05a100 con 0x7fe6e409af10 2024-04-15T15:22:03.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.067+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe6e40cf470 con 0x7fe6e409af10 2024-04-15T15:22:03.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.067+0000 7fe6e17fa640 1 -- 172.21.15.12:0/3557183020 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fe6dc05e400 con 0x7fe6e409af10 2024-04-15T15:22:03.069 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c920 msgr2=0x7fe6d006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c920 0x7fe6d006ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8036000 tx=0x7fe6d8008030 comp rx=0 tx=0).stop 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409af10 msgr2=0x7fe6e4158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409af10 0x7fe6e4158b50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00cf60 tx=0x7fe6dc00cf90 comp rx=0 tx=0).stop 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 shutdown_connections 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c920 0x7fe6d006ede0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6e406dd40 0x7fe6e40710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6e409b650 0x7fe6e406d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 --2- 172.21.15.12:0/3557183020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e409af10 0x7fe6e4158b50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 >> 172.21.15.12:0/3557183020 conn(0x7fe6e40b3fb0 msgr2=0x7fe6e415b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 shutdown_connections 2024-04-15T15:22:03.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.069+0000 7fe6eaff2640 1 -- 172.21.15.12:0/3557183020 wait complete. 2024-04-15T15:22:03.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:03.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.186+0000 7ff39a700640 1 Processor -- start 2024-04-15T15:22:03.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.186+0000 7ff39a700640 1 -- start start 2024-04-15T15:22:03.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.186+0000 7ff39a700640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff39415e700 0x7ff3940b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.186+0000 7ff39a700640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3940b47a0 con 0x7ff3940d3270 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.186+0000 7ff39a700640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3940b4a80 con 0x7ff3940d2360 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.186+0000 7ff39a700640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff3940b53b0 con 0x7ff39415e700 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.187+0000 7ff393fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff39415e700 0x7ff3940b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.187+0000 7ff398c9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff3940d3270 0x7ff3940cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34664/0 (socket says 172.21.15.12:34664) 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.187+0000 7ff398c9d640 1 -- 172.21.15.12:0/897955557 learned_addr learned my addr 172.21.15.12:0/897955557 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.187+0000 7ff393fff640 1 -- 172.21.15.12:0/897955557 >> v1:172.21.15.55:6789/0 conn(0x7ff3940d2360 legacy=0x7ff3940d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.187+0000 7ff393fff640 1 -- 172.21.15.12:0/897955557 >> v1:172.21.15.12:6789/0 conn(0x7ff3940d3270 legacy=0x7ff3940cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.187+0000 7ff393fff640 1 -- 172.21.15.12:0/897955557 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3940b4d60 con 0x7ff39415e700 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.187+0000 7ff3937fe640 1 -- 172.21.15.12:0/897955557 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3099529635 0 0) 0x7ff3940b47a0 con 0x7ff3940d3270 2024-04-15T15:22:03.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff393fff640 1 --2- 172.21.15.12:0/897955557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39415e700 0x7ff3940b5840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff3940b5530 tx=0x7ff38402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ae7b29417954a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff3937fe640 1 -- 172.21.15.12:0/897955557 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384035020 con 0x7ff39415e700 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff3937fe640 1 -- 172.21.15.12:0/897955557 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff384031800 con 0x7ff39415e700 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff3937fe640 1 -- 172.21.15.12:0/897955557 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384031bd0 con 0x7ff39415e700 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff39a700640 1 -- 172.21.15.12:0/897955557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39415e700 msgr2=0x7ff3940b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff39a700640 1 --2- 172.21.15.12:0/897955557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39415e700 0x7ff3940b5840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff3940b5530 tx=0x7ff38402b8a0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff39a700640 1 -- 172.21.15.12:0/897955557 shutdown_connections 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff39a700640 1 --2- 172.21.15.12:0/897955557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39415e700 0x7ff3940b5840 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff39a700640 1 -- 172.21.15.12:0/897955557 >> 172.21.15.12:0/897955557 conn(0x7ff39408bf60 msgr2=0x7ff3940d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff39a700640 1 -- 172.21.15.12:0/897955557 shutdown_connections 2024-04-15T15:22:03.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.188+0000 7ff39a700640 1 -- 172.21.15.12:0/897955557 wait complete. 2024-04-15T15:22:03.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff39a700640 1 Processor -- start 2024-04-15T15:22:03.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff39a700640 1 -- start start 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff39a700640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff3940d2360 0x7ff3941509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff39a700640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3940d3270 0x7ff3940cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff39a700640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff39415e700 0x7ff3940b8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff39a700640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff3940b5d80 con 0x7ff3940d3270 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff39a700640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff3940b4450 con 0x7ff3940d2360 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff39a700640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff3940d1d40 con 0x7ff39415e700 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff38bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3940d3270 0x7ff3940cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff38bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3940d3270 0x7ff3940cdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52752/0 (socket says 172.21.15.12:52752) 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.190+0000 7ff38bfff640 1 -- 172.21.15.12:0/1205502918 learned_addr learned my addr 172.21.15.12:0/1205502918 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:03.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff393fff640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff3940d2360 0x7ff3941509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff398c9d640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff39415e700 0x7ff3940b8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff38bfff640 1 -- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff39415e700 msgr2=0x7ff3940b8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff38bfff640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff39415e700 0x7ff3940b8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff38bfff640 1 -- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff3940d2360 msgr2=0x7ff3941509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff38bfff640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff3940d2360 0x7ff3941509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff38bfff640 1 -- 172.21.15.12:0/1205502918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3940cc270 con 0x7ff3940d3270 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff398c9d640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff39415e700 0x7ff3940b8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff38bfff640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3940d3270 0x7ff3940cdd80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00c410 tx=0x7ff37c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff3917fa640 1 -- 172.21.15.12:0/1205502918 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c017070 con 0x7ff3940d3270 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff3917fa640 1 -- 172.21.15.12:0/1205502918 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff37c0035f0 con 0x7ff3940d3270 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3940cc550 con 0x7ff3940d3270 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff3917fa640 1 -- 172.21.15.12:0/1205502918 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c003a20 con 0x7ff3940d3270 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.191+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3940ccb10 con 0x7ff3940d3270 2024-04-15T15:22:03.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.192+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3940ccd10 con 0x7ff3940d3270 2024-04-15T15:22:03.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.193+0000 7ff3917fa640 1 -- 172.21.15.12:0/1205502918 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff37c01a070 con 0x7ff3940d3270 2024-04-15T15:22:03.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.193+0000 7ff3917fa640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff37406c8f0 0x7ff37406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.193+0000 7ff3917fa640 1 -- 172.21.15.12:0/1205502918 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7ff37c08b3b0 con 0x7ff3940d3270 2024-04-15T15:22:03.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.193+0000 7ff393fff640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff37406c8f0 0x7ff37406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.194+0000 7ff393fff640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff37406c8f0 0x7ff37406edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff384036000 tx=0x7ff38403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.198+0000 7ff3917fa640 1 -- 172.21.15.12:0/1205502918 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff3940ccd10 con 0x7ff3940d3270 2024-04-15T15:22:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.398+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff3940689e0 con 0x7ff3940d3270 2024-04-15T15:22:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.398+0000 7ff3917fa640 1 -- 172.21.15.12:0/1205502918 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7ff37c059430 con 0x7ff3940d3270 2024-04-15T15:22:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff37406c8f0 msgr2=0x7ff37406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff37406c8f0 0x7ff37406edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff384036000 tx=0x7ff38403c040 comp rx=0 tx=0).stop 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3940d3270 msgr2=0x7ff3940cdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3940d3270 0x7ff3940cdd80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00c410 tx=0x7ff37c00c8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 shutdown_connections 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff39415e700 0x7ff3940b8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3940d3270 0x7ff3940cdd80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff37406c8f0 0x7ff37406edb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 --2- 172.21.15.12:0/1205502918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff3940d2360 0x7ff3941509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 >> 172.21.15.12:0/1205502918 conn(0x7ff39408bf60 msgr2=0x7ff3940cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 shutdown_connections 2024-04-15T15:22:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.400+0000 7ff39a700640 1 -- 172.21.15.12:0/1205502918 wait complete. 2024-04-15T15:22:03.412 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:22:03.412 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:03.428 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:03.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.572+0000 7f2e078ba640 1 Processor -- start 2024-04-15T15:22:03.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.572+0000 7f2e078ba640 1 -- start start 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e078ba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2e000b3160 0x7f2e0015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e078ba640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e000ae280 0x7f2e000ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e078ba640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e0015cb50 con 0x7f2e000d0690 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e078ba640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2e0015bfd0 con 0x7f2e000ae280 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e078ba640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2e0015c2b0 con 0x7f2e000b3160 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e05e57640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e000ae280 0x7f2e000ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e05e57640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2e000ae280 0x7f2e000ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52768/0 (socket says 172.21.15.12:52768) 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e05e57640 1 -- 172.21.15.12:0/2527630929 learned_addr learned my addr 172.21.15.12:0/2527630929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e04e55640 1 --2- 172.21.15.12:0/2527630929 >> v2:172.21.15.55:3300/0 conn(0x7f2e000b3160 0x7f2e0015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e05e57640 1 -- 172.21.15.12:0/2527630929 >> v2:172.21.15.55:3300/0 conn(0x7f2e000b3160 msgr2=0x7f2e0015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e05e57640 1 --2- 172.21.15.12:0/2527630929 >> v2:172.21.15.55:3300/0 conn(0x7f2e000b3160 0x7f2e0015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e05e57640 1 -- 172.21.15.12:0/2527630929 >> v1:172.21.15.55:6789/0 conn(0x7f2e000d0690 legacy=0x7f2e000d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:03.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.573+0000 7f2e05e57640 1 -- 172.21.15.12:0/2527630929 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0015c590 con 0x7f2e000ae280 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.574+0000 7f2e05e57640 1 --2- 172.21.15.12:0/2527630929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e000ae710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2e0006fec0 tx=0x7f2df802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d1b99b0a9db6b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.574+0000 7f2df7fff640 1 -- 172.21.15.12:0/2527630929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df802ca80 con 0x7f2e000ae280 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.574+0000 7f2df7fff640 1 -- 172.21.15.12:0/2527630929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2df802cc20 con 0x7f2e000ae280 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.574+0000 7f2df7fff640 1 -- 172.21.15.12:0/2527630929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df80343f0 con 0x7f2e000ae280 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.575+0000 7f2e078ba640 1 -- 172.21.15.12:0/2527630929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 msgr2=0x7f2e000ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.575+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2527630929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e000ae710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2e0006fec0 tx=0x7f2df802bee0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.575+0000 7f2e078ba640 1 -- 172.21.15.12:0/2527630929 shutdown_connections 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.575+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2527630929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e000ae710 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.575+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2527630929 >> v2:172.21.15.55:3300/0 conn(0x7f2e000b3160 0x7f2e0015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.575+0000 7f2e078ba640 1 -- 172.21.15.12:0/2527630929 >> 172.21.15.12:0/2527630929 conn(0x7f2e0006be00 msgr2=0x7f2e0006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:03.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.575+0000 7f2e078ba640 1 -- 172.21.15.12:0/2527630929 shutdown_connections 2024-04-15T15:22:03.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.575+0000 7f2e078ba640 1 -- 172.21.15.12:0/2527630929 wait complete. 2024-04-15T15:22:03.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.576+0000 7f2e078ba640 1 Processor -- start 2024-04-15T15:22:03.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.576+0000 7f2e078ba640 1 -- start start 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e078ba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e00147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e078ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e000b3160 0x7f2e00157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e078ba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e000d0690 0x7f2e001468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e078ba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0015bbd0 con 0x7f2e000ae280 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e078ba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2e000b4fb0 con 0x7f2e000b3160 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e078ba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2e0015bd50 con 0x7f2e000d0690 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e00147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e00147670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52782/0 (socket says 172.21.15.12:52782) 2024-04-15T15:22:03.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 -- 172.21.15.12:0/2524045697 learned_addr learned my addr 172.21.15.12:0/2524045697 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e04e55640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e000b3160 0x7f2e00157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 -- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e000d0690 msgr2=0x7f2e001468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e000d0690 0x7f2e001468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 -- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e000b3160 msgr2=0x7f2e00157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e000b3160 0x7f2e00157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 -- 172.21.15.12:0/2524045697 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e00145830 con 0x7f2e000ae280 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e04e55640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e000b3160 0x7f2e00157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e05656640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e00147670 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00b3f0 tx=0x7f2dfc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2df5ffb640 1 -- 172.21.15.12:0/2524045697 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc015020 con 0x7f2e000ae280 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2df5ffb640 1 -- 172.21.15.12:0/2524045697 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dfc011830 con 0x7f2e000ae280 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e00145b10 con 0x7f2e000ae280 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.577+0000 7f2df5ffb640 1 -- 172.21.15.12:0/2524045697 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc011c20 con 0x7f2e000ae280 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.578+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e001460d0 con 0x7f2e000ae280 2024-04-15T15:22:03.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.579+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e000cfec0 con 0x7f2e000ae280 2024-04-15T15:22:03.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.579+0000 7f2df5ffb640 1 -- 172.21.15.12:0/2524045697 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2dfc018020 con 0x7f2e000ae280 2024-04-15T15:22:03.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.579+0000 7f2df5ffb640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2de406c920 0x7f2de406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.579+0000 7f2df5ffb640 1 -- 172.21.15.12:0/2524045697 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f2dfc08bbd0 con 0x7f2e000ae280 2024-04-15T15:22:03.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.579+0000 7f2e04e55640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2de406c920 0x7f2de406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.580+0000 7f2e04e55640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2de406c920 0x7f2de406ede0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2df0006fd0 tx=0x7f2df0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.584+0000 7f2df5ffb640 1 -- 172.21.15.12:0/2524045697 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e000cfec0 con 0x7f2e000ae280 2024-04-15T15:22:03.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.770+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e000689e0 con 0x7f2e000ae280 2024-04-15T15:22:03.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.772+0000 7f2df5ffb640 1 -- 172.21.15.12:0/2524045697 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f2dfc01f030 con 0x7f2e000ae280 2024-04-15T15:22:03.773 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:03.773 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:03.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.773+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2de406c920 msgr2=0x7f2de406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.773+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2de406c920 0x7f2de406ede0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2df0006fd0 tx=0x7f2df0008040 comp rx=0 tx=0).stop 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.773+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 msgr2=0x7f2e00147670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.773+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e00147670 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00b3f0 tx=0x7f2dfc00b8b0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.774+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 shutdown_connections 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.774+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2e000d0690 0x7f2e001468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.774+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2de406c920 0x7f2de406ede0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.774+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e000b3160 0x7f2e00157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.774+0000 7f2e078ba640 1 --2- 172.21.15.12:0/2524045697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e000ae280 0x7f2e00147670 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.774+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 >> 172.21.15.12:0/2524045697 conn(0x7f2e0006be00 msgr2=0x7f2e0006dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.774+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 shutdown_connections 2024-04-15T15:22:03.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.774+0000 7f2e078ba640 1 -- 172.21.15.12:0/2524045697 wait complete. 2024-04-15T15:22:03.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:03.787 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.893+0000 7f3e8c10e640 1 Processor -- start 2024-04-15T15:22:03.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.893+0000 7f3e8c10e640 1 -- start start 2024-04-15T15:22:03.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e8c10e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e8415e790 0x7f3e840b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e8c10e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3e840d1cf0 0x7f3e840d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e8c10e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e8415eb70 con 0x7f3e840b7be0 2024-04-15T15:22:03.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e8c10e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3e840d3790 con 0x7f3e840d1cf0 2024-04-15T15:22:03.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e8c10e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3e840d45b0 con 0x7f3e8415e790 2024-04-15T15:22:03.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e8a6ab640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3e840d1cf0 0x7f3e840d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e89eaa640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e8415e790 0x7f3e840b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e896a9640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3e840b7be0 0x7f3e840d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:60276/0 (socket says 172.21.15.12:60276) 2024-04-15T15:22:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.894+0000 7f3e896a9640 1 -- 172.21.15.12:0/1753173840 learned_addr learned my addr 172.21.15.12:0/1753173840 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.895+0000 7f3e8a6ab640 1 -- 172.21.15.12:0/1753173840 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e8415e790 msgr2=0x7f3e840b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.895+0000 7f3e8a6ab640 1 --2- 172.21.15.12:0/1753173840 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e8415e790 0x7f3e840b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.895+0000 7f3e8a6ab640 1 -- 172.21.15.12:0/1753173840 >> v1:172.21.15.55:6789/0 conn(0x7f3e840b7be0 legacy=0x7f3e840d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.895+0000 7f3e8a6ab640 1 -- 172.21.15.12:0/1753173840 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e840d3c80 con 0x7f3e840d1cf0 2024-04-15T15:22:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.895+0000 7f3e88ea8640 1 -- 172.21.15.12:0/1753173840 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2125957808 0 0) 0x7f3e8415eb70 con 0x7f3e840b7be0 2024-04-15T15:22:03.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.895+0000 7f3e89eaa640 1 --2- 172.21.15.12:0/1753173840 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e8415e790 0x7f3e840b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.895+0000 7f3e8a6ab640 1 --2- 172.21.15.12:0/1753173840 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e840d1cf0 0x7f3e840d4a40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3e840d3910 tx=0x7f3e7c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20208c28f121e175 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e88ea8640 1 -- 172.21.15.12:0/1753173840 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c039070 con 0x7f3e840d1cf0 2024-04-15T15:22:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e88ea8640 1 -- 172.21.15.12:0/1753173840 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e7c02cb20 con 0x7f3e840d1cf0 2024-04-15T15:22:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e88ea8640 1 -- 172.21.15.12:0/1753173840 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c034440 con 0x7f3e840d1cf0 2024-04-15T15:22:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e8c10e640 1 -- 172.21.15.12:0/1753173840 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e840d1cf0 msgr2=0x7f3e840d4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/1753173840 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e840d1cf0 0x7f3e840d4a40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3e840d3910 tx=0x7f3e7c02bd70 comp rx=0 tx=0).stop 2024-04-15T15:22:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e8c10e640 1 -- 172.21.15.12:0/1753173840 shutdown_connections 2024-04-15T15:22:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/1753173840 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e840d1cf0 0x7f3e840d4a40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/1753173840 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e8415e790 0x7f3e840b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e8c10e640 1 -- 172.21.15.12:0/1753173840 >> 172.21.15.12:0/1753173840 conn(0x7f3e84067b40 msgr2=0x7f3e84067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e8c10e640 1 -- 172.21.15.12:0/1753173840 shutdown_connections 2024-04-15T15:22:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.896+0000 7f3e8c10e640 1 -- 172.21.15.12:0/1753173840 wait complete. 2024-04-15T15:22:03.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.897+0000 7f3e8c10e640 1 Processor -- start 2024-04-15T15:22:03.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.897+0000 7f3e8c10e640 1 -- start start 2024-04-15T15:22:03.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e8c10e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e840b7be0 0x7f3e840c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e8c10e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e840d1cf0 0x7f3e841587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e8c10e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e8415e790 0x7f3e8415c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e8c10e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3e840d4f80 con 0x7f3e840d1cf0 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e8c10e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3e840d4240 con 0x7f3e8415e790 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e8c10e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3e840d5100 con 0x7f3e840b7be0 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e896a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e840d1cf0 0x7f3e841587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e896a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e840d1cf0 0x7f3e841587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52788/0 (socket says 172.21.15.12:52788) 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e896a9640 1 -- 172.21.15.12:0/3610028280 learned_addr learned my addr 172.21.15.12:0/3610028280 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.898+0000 7f3e8a6ab640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e8415e790 0x7f3e8415c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e89eaa640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e840b7be0 0x7f3e840c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e896a9640 1 -- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e840b7be0 msgr2=0x7f3e840c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e896a9640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e840b7be0 0x7f3e840c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e896a9640 1 -- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e8415e790 msgr2=0x7f3e8415c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e896a9640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e8415e790 0x7f3e8415c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e896a9640 1 -- 172.21.15.12:0/3610028280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e840c9eb0 con 0x7f3e840d1cf0 2024-04-15T15:22:03.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e89eaa640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e840b7be0 0x7f3e840c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:03.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e896a9640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e840d1cf0 0x7f3e841587a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400c4a0 tx=0x7f3e7400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e7a7fc640 1 -- 172.21.15.12:0/3610028280 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74017070 con 0x7f3e840d1cf0 2024-04-15T15:22:03.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e840ca2e0 con 0x7f3e840d1cf0 2024-04-15T15:22:03.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e7a7fc640 1 -- 172.21.15.12:0/3610028280 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e74005ce0 con 0x7f3e840d1cf0 2024-04-15T15:22:03.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e7a7fc640 1 -- 172.21.15.12:0/3610028280 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e740072b0 con 0x7f3e840d1cf0 2024-04-15T15:22:03.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.899+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e840ca860 con 0x7f3e840d1cf0 2024-04-15T15:22:03.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.900+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e840689e0 con 0x7f3e840d1cf0 2024-04-15T15:22:03.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.901+0000 7f3e7a7fc640 1 -- 172.21.15.12:0/3610028280 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e7401a070 con 0x7f3e840d1cf0 2024-04-15T15:22:03.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.901+0000 7f3e7a7fc640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e6806c8f0 0x7f3e6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:03.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.902+0000 7f3e7a7fc640 1 -- 172.21.15.12:0/3610028280 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f3e7408b360 con 0x7f3e840d1cf0 2024-04-15T15:22:03.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.902+0000 7f3e89eaa640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e6806c8f0 0x7f3e6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:03.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.903+0000 7f3e89eaa640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e6806c8f0 0x7f3e6806edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009a20 tx=0x7f3e80008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:03.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:03.906+0000 7f3e7a7fc640 1 -- 172.21.15.12:0/3610028280 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e74059360 con 0x7f3e840d1cf0 2024-04-15T15:22:04.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.091+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e840d1780 con 0x7f3e840d1cf0 2024-04-15T15:22:04.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.092+0000 7f3e7a7fc640 1 -- 172.21.15.12:0/3610028280 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f3e74058c60 con 0x7f3e840d1cf0 2024-04-15T15:22:04.093 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:04.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e6806c8f0 msgr2=0x7f3e6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e6806c8f0 0x7f3e6806edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009a20 tx=0x7f3e80008040 comp rx=0 tx=0).stop 2024-04-15T15:22:04.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e840d1cf0 msgr2=0x7f3e841587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e840d1cf0 0x7f3e841587a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400c4a0 tx=0x7f3e7400c960 comp rx=0 tx=0).stop 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 shutdown_connections 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3e6806c8f0 0x7f3e6806edb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e8415e790 0x7f3e8415c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e840d1cf0 0x7f3e841587a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 --2- 172.21.15.12:0/3610028280 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3e840b7be0 0x7f3e840c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 >> 172.21.15.12:0/3610028280 conn(0x7f3e84067b40 msgr2=0x7f3e840d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 shutdown_connections 2024-04-15T15:22:04.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.094+0000 7f3e8c10e640 1 -- 172.21.15.12:0/3610028280 wait complete. 2024-04-15T15:22:04.106 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.213+0000 7f1220293640 1 Processor -- start 2024-04-15T15:22:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.213+0000 7f1220293640 1 -- start start 2024-04-15T15:22:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f1220293640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f121806f6d0 0x7f12180b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f1220293640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f121815f5c0 0x7f12180729b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f1220293640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f121806fab0 con 0x7f12180d0570 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f1220293640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f12180b49a0 con 0x7f121806f6d0 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f1220293640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f12180b43e0 con 0x7f121815f5c0 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121e02f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f121806f6d0 0x7f12180b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121e830640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f12180d0570 0x7f121806e5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34680/0 (socket says 172.21.15.12:34680) 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121e830640 1 -- 172.21.15.12:0/2478437748 learned_addr learned my addr 172.21.15.12:0/2478437748 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121e02f640 1 --2- 172.21.15.12:0/2478437748 >> v2:172.21.15.12:3300/0 conn(0x7f121806f6d0 0x7f12180b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52794/0 (socket says 172.21.15.12:52794) 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121d82e640 1 --2- 172.21.15.12:0/2478437748 >> v2:172.21.15.55:3300/0 conn(0x7f121815f5c0 0x7f12180729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121e02f640 1 -- 172.21.15.12:0/2478437748 >> v2:172.21.15.55:3300/0 conn(0x7f121815f5c0 msgr2=0x7f12180729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121e02f640 1 --2- 172.21.15.12:0/2478437748 >> v2:172.21.15.55:3300/0 conn(0x7f121815f5c0 0x7f12180729b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121e02f640 1 -- 172.21.15.12:0/2478437748 >> v1:172.21.15.12:6789/0 conn(0x7f12180d0570 legacy=0x7f121806e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121d02d640 1 -- 172.21.15.12:0/2478437748 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2759827100 0 0) 0x7f1210001490 con 0x7f12180d0570 2024-04-15T15:22:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.214+0000 7f121e02f640 1 -- 172.21.15.12:0/2478437748 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12180b46c0 con 0x7f121806f6d0 2024-04-15T15:22:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.215+0000 7f121e02f640 1 --2- 172.21.15.12:0/2478437748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f121806f6d0 0x7f12180b3e00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f12180ad840 tx=0x7f121402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9fff6165f1855b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.215+0000 7f121d02d640 1 -- 172.21.15.12:0/2478437748 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214035020 con 0x7f121806f6d0 2024-04-15T15:22:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.215+0000 7f121d02d640 1 -- 172.21.15.12:0/2478437748 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1214031830 con 0x7f121806f6d0 2024-04-15T15:22:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.215+0000 7f121d02d640 1 -- 172.21.15.12:0/2478437748 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214031c00 con 0x7f121806f6d0 2024-04-15T15:22:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.216+0000 7f1220293640 1 -- 172.21.15.12:0/2478437748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f121806f6d0 msgr2=0x7f12180b3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.216+0000 7f1220293640 1 --2- 172.21.15.12:0/2478437748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f121806f6d0 0x7f12180b3e00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f12180ad840 tx=0x7f121402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.216+0000 7f1220293640 1 -- 172.21.15.12:0/2478437748 shutdown_connections 2024-04-15T15:22:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.216+0000 7f1220293640 1 --2- 172.21.15.12:0/2478437748 >> v2:172.21.15.55:3300/0 conn(0x7f121815f5c0 0x7f12180729b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.216+0000 7f1220293640 1 --2- 172.21.15.12:0/2478437748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f121806f6d0 0x7f12180b3e00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.216+0000 7f1220293640 1 -- 172.21.15.12:0/2478437748 >> 172.21.15.12:0/2478437748 conn(0x7f121806b9e0 msgr2=0x7f121806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.216+0000 7f1220293640 1 -- 172.21.15.12:0/2478437748 shutdown_connections 2024-04-15T15:22:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.216+0000 7f1220293640 1 -- 172.21.15.12:0/2478437748 wait complete. 2024-04-15T15:22:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.217+0000 7f1220293640 1 Processor -- start 2024-04-15T15:22:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.217+0000 7f1220293640 1 -- start start 2024-04-15T15:22:04.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f1220293640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f121806f6d0 0x7f121814f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f1220293640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12180d0570 0x7f12181543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f1220293640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f121815f5c0 0x7f1218150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f1220293640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f121815fa70 con 0x7f12180d0570 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f1220293640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1218072490 con 0x7f121815f5c0 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f1220293640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1218072ef0 con 0x7f121806f6d0 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12180d0570 0x7f12181543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12180d0570 0x7f12181543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52798/0 (socket says 172.21.15.12:52798) 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 -- 172.21.15.12:0/1158348521 learned_addr learned my addr 172.21.15.12:0/1158348521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 -- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f121806f6d0 msgr2=0x7f121814f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121e02f640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f121806f6d0 0x7f121814f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121e830640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f121815f5c0 0x7f1218150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f121806f6d0 0x7f121814f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 -- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f121815f5c0 msgr2=0x7f1218150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f121815f5c0 0x7f1218150aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 -- 172.21.15.12:0/1158348521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12180cea90 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f121d82e640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12180d0570 0x7f12181543b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f120800c820 tx=0x7f120800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f120effd640 1 -- 172.21.15.12:0/1158348521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208015070 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f120effd640 1 -- 172.21.15.12:0/1158348521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1208004020 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121814fce0 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f120effd640 1 -- 172.21.15.12:0/1158348521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12080043f0 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.218+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12181502a0 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.219+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12180689e0 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.220+0000 7f120effd640 1 -- 172.21.15.12:0/1158348521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1208007830 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.220+0000 7f120effd640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11fc06c8f0 0x7f11fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.220+0000 7f120effd640 1 -- 172.21.15.12:0/1158348521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f120808bcb0 con 0x7f12180d0570 2024-04-15T15:22:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.220+0000 7f121e02f640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11fc06c8f0 0x7f11fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.221+0000 7f121e02f640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11fc06c8f0 0x7f11fc06edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1214036000 tx=0x7f1214008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.225+0000 7f120effd640 1 -- 172.21.15.12:0/1158348521 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1208059d30 con 0x7f12180d0570 2024-04-15T15:22:04.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.407+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1218161110 con 0x7f12180d0570 2024-04-15T15:22:04.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.408+0000 7f120effd640 1 -- 172.21.15.12:0/1158348521 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f1208059630 con 0x7f12180d0570 2024-04-15T15:22:04.409 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:04.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.409+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11fc06c8f0 msgr2=0x7f11fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.409+0000 7f1220293640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11fc06c8f0 0x7f11fc06edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1214036000 tx=0x7f1214008030 comp rx=0 tx=0).stop 2024-04-15T15:22:04.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.409+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12180d0570 msgr2=0x7f12181543b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.409+0000 7f1220293640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12180d0570 0x7f12181543b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f120800c820 tx=0x7f120800cce0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.410+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 shutdown_connections 2024-04-15T15:22:04.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.410+0000 7f1220293640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f11fc06c8f0 0x7f11fc06edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.410+0000 7f1220293640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f121815f5c0 0x7f1218150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.410+0000 7f1220293640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12180d0570 0x7f12181543b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.410+0000 7f1220293640 1 --2- 172.21.15.12:0/1158348521 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f121806f6d0 0x7f121814f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.410+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 >> 172.21.15.12:0/1158348521 conn(0x7f121806b9e0 msgr2=0x7f1218161630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:04.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.410+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 shutdown_connections 2024-04-15T15:22:04.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.410+0000 7f1220293640 1 -- 172.21.15.12:0/1158348521 wait complete. 2024-04-15T15:22:04.422 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:04.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.527+0000 7ff4bbf8f640 1 Processor -- start 2024-04-15T15:22:04.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.527+0000 7ff4bbf8f640 1 -- start start 2024-04-15T15:22:04.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4bbf8f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 0x7ff4b40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4bbf8f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4b40d2360 0x7ff4b40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4bbf8f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff4b415eae0 0x7ff4b40d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4bbf8f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b40b53b0 con 0x7ff4b40d2360 2024-04-15T15:22:04.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4bbf8f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b40b47a0 con 0x7ff4b415eae0 2024-04-15T15:22:04.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4bbf8f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b40b4a80 con 0x7ff4b415e700 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4b952a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4b40d2360 0x7ff4b40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4b952a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4b40d2360 0x7ff4b40d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52816/0 (socket says 172.21.15.12:52816) 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.528+0000 7ff4b952a640 1 -- 172.21.15.12:0/1503418311 learned_addr learned my addr 172.21.15.12:0/1503418311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.529+0000 7ff4b9d2b640 1 --2- 172.21.15.12:0/1503418311 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 0x7ff4b40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.529+0000 7ff4ba52c640 1 --2- 172.21.15.12:0/1503418311 >> v2:172.21.15.55:3300/0 conn(0x7ff4b415eae0 0x7ff4b40d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.529+0000 7ff4b952a640 1 -- 172.21.15.12:0/1503418311 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 msgr2=0x7ff4b40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.529+0000 7ff4b952a640 1 --2- 172.21.15.12:0/1503418311 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 0x7ff4b40b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.529+0000 7ff4b952a640 1 -- 172.21.15.12:0/1503418311 >> v2:172.21.15.55:3300/0 conn(0x7ff4b415eae0 msgr2=0x7ff4b40d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.529+0000 7ff4b952a640 1 --2- 172.21.15.12:0/1503418311 >> v2:172.21.15.55:3300/0 conn(0x7ff4b415eae0 0x7ff4b40d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.529+0000 7ff4b952a640 1 -- 172.21.15.12:0/1503418311 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b40b4d60 con 0x7ff4b40d2360 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.529+0000 7ff4b952a640 1 --2- 172.21.15.12:0/1503418311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b40d2360 0x7ff4b40d2b60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff4b408c780 tx=0x7ff4a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334b62e76767c555 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4b8d29640 1 -- 172.21.15.12:0/1503418311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4035020 con 0x7ff4b40d2360 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4b8d29640 1 -- 172.21.15.12:0/1503418311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a4031870 con 0x7ff4b40d2360 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4b8d29640 1 -- 172.21.15.12:0/1503418311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4031c40 con 0x7ff4b40d2360 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/1503418311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b40d2360 msgr2=0x7ff4b40d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/1503418311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b40d2360 0x7ff4b40d2b60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff4b408c780 tx=0x7ff4a402b910 comp rx=0 tx=0).stop 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/1503418311 shutdown_connections 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/1503418311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b40d2360 0x7ff4b40d2b60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/1503418311 >> v2:172.21.15.55:3300/0 conn(0x7ff4b415eae0 0x7ff4b40d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/1503418311 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 0x7ff4b40b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/1503418311 >> 172.21.15.12:0/1503418311 conn(0x7ff4b408bf60 msgr2=0x7ff4b40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/1503418311 shutdown_connections 2024-04-15T15:22:04.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.530+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/1503418311 wait complete. 2024-04-15T15:22:04.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.531+0000 7ff4bbf8f640 1 Processor -- start 2024-04-15T15:22:04.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4bbf8f640 1 -- start start 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4bbf8f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4b40d2360 0x7ff4b415efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4bbf8f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 0x7ff4b415ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4bbf8f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b415eae0 0x7ff4b4161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4bbf8f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b40b5d80 con 0x7ff4b415eae0 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4bbf8f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b40b4450 con 0x7ff4b40d2360 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4bbf8f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b40d1d40 con 0x7ff4b415e700 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4ba52c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b415eae0 0x7ff4b4161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4ba52c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b415eae0 0x7ff4b4161120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52818/0 (socket says 172.21.15.12:52818) 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.532+0000 7ff4ba52c640 1 -- 172.21.15.12:0/3743999744 learned_addr learned my addr 172.21.15.12:0/3743999744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4b9d2b640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4b40d2360 0x7ff4b415efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4ba52c640 1 -- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 msgr2=0x7ff4b415ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4b952a640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 0x7ff4b415ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4ba52c640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 0x7ff4b415ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4ba52c640 1 -- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4b40d2360 msgr2=0x7ff4b415efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4ba52c640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4b40d2360 0x7ff4b415efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4ba52c640 1 -- 172.21.15.12:0/3743999744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b415f5b0 con 0x7ff4b415eae0 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4ba52c640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b415eae0 0x7ff4b4161120 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac007e40 tx=0x7ff4ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4aa7fc640 1 -- 172.21.15.12:0/3743999744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac014620 con 0x7ff4b415eae0 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4aa7fc640 1 -- 172.21.15.12:0/3743999744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4ac019070 con 0x7ff4b415eae0 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b415f890 con 0x7ff4b415eae0 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4aa7fc640 1 -- 172.21.15.12:0/3743999744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac014a20 con 0x7ff4b415eae0 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.533+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b415fe50 con 0x7ff4b415eae0 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.534+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b40689e0 con 0x7ff4b415eae0 2024-04-15T15:22:04.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.535+0000 7ff4aa7fc640 1 -- 172.21.15.12:0/3743999744 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff4ac014bc0 con 0x7ff4b415eae0 2024-04-15T15:22:04.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.535+0000 7ff4aa7fc640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff49806c8f0 0x7ff49806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.535+0000 7ff4aa7fc640 1 -- 172.21.15.12:0/3743999744 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7ff4ac08c8a0 con 0x7ff4b415eae0 2024-04-15T15:22:04.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.535+0000 7ff4b9d2b640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff49806c8f0 0x7ff49806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.536+0000 7ff4b9d2b640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff49806c8f0 0x7ff49806edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff4b00096f0 tx=0x7ff4b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.539+0000 7ff4aa7fc640 1 -- 172.21.15.12:0/3743999744 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4ac05a890 con 0x7ff4b415eae0 2024-04-15T15:22:04.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.725+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff4b40ba500 con 0x7ff4b415eae0 2024-04-15T15:22:04.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.726+0000 7ff4aa7fc640 1 -- 172.21.15.12:0/3743999744 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7ff4ac05a190 con 0x7ff4b415eae0 2024-04-15T15:22:04.727 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:04.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.727+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff49806c8f0 msgr2=0x7ff49806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.727+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff49806c8f0 0x7ff49806edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff4b00096f0 tx=0x7ff4b0008040 comp rx=0 tx=0).stop 2024-04-15T15:22:04.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.727+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b415eae0 msgr2=0x7ff4b4161120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.727+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b415eae0 0x7ff4b4161120 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac007e40 tx=0x7ff4ac00e4b0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.728+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 shutdown_connections 2024-04-15T15:22:04.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.728+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4b415eae0 0x7ff4b4161120 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.728+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4b415e700 0x7ff4b415ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.728+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff49806c8f0 0x7ff49806edb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.728+0000 7ff4bbf8f640 1 --2- 172.21.15.12:0/3743999744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4b40d2360 0x7ff4b415efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.728+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 >> 172.21.15.12:0/3743999744 conn(0x7ff4b408bf60 msgr2=0x7ff4b40d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:04.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.728+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 shutdown_connections 2024-04-15T15:22:04.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.728+0000 7ff4bbf8f640 1 -- 172.21.15.12:0/3743999744 wait complete. 2024-04-15T15:22:04.740 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.846+0000 7f4038740640 1 Processor -- start 2024-04-15T15:22:04.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.846+0000 7f4038740640 1 -- start start 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4038740640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f403015e700 0x7f40300b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4038740640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f40300d3270 0x7f40300cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4038740640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40300b47a0 con 0x7f40300d2360 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4038740640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f40300b53b0 con 0x7f403015e700 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4038740640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f40300b4a80 con 0x7f40300d3270 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4036cdd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f40300d3270 0x7f40300cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f40364dc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f403015e700 0x7f40300b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4035cdb640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f40300d2360 0x7f40300d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:60290/0 (socket says 172.21.15.12:60290) 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4035cdb640 1 -- 172.21.15.12:0/3015651466 learned_addr learned my addr 172.21.15.12:0/3015651466 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:04.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f40354da640 1 -- 172.21.15.12:0/3015651466 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793679987 0 0) 0x7f40300b47a0 con 0x7f40300d2360 2024-04-15T15:22:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f40354da640 1 -- 172.21.15.12:0/3015651466 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f401c003520 con 0x7f40300d2360 2024-04-15T15:22:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4036cdd640 1 -- 172.21.15.12:0/3015651466 >> v2:172.21.15.55:3300/0 conn(0x7f403015e700 msgr2=0x7f40300b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4036cdd640 1 --2- 172.21.15.12:0/3015651466 >> v2:172.21.15.55:3300/0 conn(0x7f403015e700 0x7f40300b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.847+0000 7f4036cdd640 1 -- 172.21.15.12:0/3015651466 >> v1:172.21.15.55:6789/0 conn(0x7f40300d2360 legacy=0x7f40300d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.848+0000 7f4036cdd640 1 -- 172.21.15.12:0/3015651466 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40300b4d60 con 0x7f40300d3270 2024-04-15T15:22:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.848+0000 7f40364dc640 1 --2- 172.21.15.12:0/3015651466 >> v2:172.21.15.55:3300/0 conn(0x7f403015e700 0x7f40300b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:04.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.848+0000 7f4036cdd640 1 --2- 172.21.15.12:0/3015651466 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f40300d3270 0x7f40300cf370 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4028009d90 tx=0x7f402802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78e66f45dd8ec07e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f40354da640 1 -- 172.21.15.12:0/3015651466 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028039070 con 0x7f40300d3270 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f40354da640 1 -- 172.21.15.12:0/3015651466 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402802cc50 con 0x7f40300d3270 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f40354da640 1 -- 172.21.15.12:0/3015651466 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028034440 con 0x7f40300d3270 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f4038740640 1 -- 172.21.15.12:0/3015651466 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f40300d3270 msgr2=0x7f40300cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f4038740640 1 --2- 172.21.15.12:0/3015651466 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f40300d3270 0x7f40300cf370 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4028009d90 tx=0x7f402802bea0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f4038740640 1 -- 172.21.15.12:0/3015651466 shutdown_connections 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f4038740640 1 --2- 172.21.15.12:0/3015651466 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f40300d3270 0x7f40300cf370 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f4038740640 1 --2- 172.21.15.12:0/3015651466 >> v2:172.21.15.55:3300/0 conn(0x7f403015e700 0x7f40300b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f4038740640 1 -- 172.21.15.12:0/3015651466 >> 172.21.15.12:0/3015651466 conn(0x7f403008bf60 msgr2=0x7f40300d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f4038740640 1 -- 172.21.15.12:0/3015651466 shutdown_connections 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.849+0000 7f4038740640 1 -- 172.21.15.12:0/3015651466 wait complete. 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.850+0000 7f4038740640 1 Processor -- start 2024-04-15T15:22:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.850+0000 7f4038740640 1 -- start start 2024-04-15T15:22:04.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f4038740640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40300d2360 0x7f403015ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f4038740640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40300d3270 0x7f40300c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f4038740640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f403015e700 0x7f403014b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f4038740640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f40300b5d80 con 0x7f40300d2360 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f4038740640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f40300b4450 con 0x7f40300d3270 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f4038740640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f40300d1d40 con 0x7f403015e700 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f40364dc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40300d2360 0x7f403015ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f40364dc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40300d2360 0x7f403015ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52838/0 (socket says 172.21.15.12:52838) 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f40364dc640 1 -- 172.21.15.12:0/2803791236 learned_addr learned my addr 172.21.15.12:0/2803791236 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f4036cdd640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f403015e700 0x7f403014b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f4035cdb640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40300d3270 0x7f40300c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f40364dc640 1 -- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f403015e700 msgr2=0x7f403014b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f40364dc640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f403015e700 0x7f403014b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f40364dc640 1 -- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40300d3270 msgr2=0x7f40300c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.851+0000 7f40364dc640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40300d3270 0x7f40300c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.852+0000 7f40364dc640 1 -- 172.21.15.12:0/2803791236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40300c9930 con 0x7f40300d2360 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.852+0000 7f4035cdb640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40300d3270 0x7f40300c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.852+0000 7f40364dc640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40300d2360 0x7f403015ed50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f402c00e9b0 tx=0x7f402c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.852+0000 7f4026ffd640 1 -- 172.21.15.12:0/2803791236 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c005990 con 0x7f40300d2360 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.852+0000 7f4026ffd640 1 -- 172.21.15.12:0/2803791236 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c005b30 con 0x7f40300d2360 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.852+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40300c9c10 con 0x7f40300d2360 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.852+0000 7f4026ffd640 1 -- 172.21.15.12:0/2803791236 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c00c9a0 con 0x7f40300d2360 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.852+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40300ca1d0 con 0x7f40300d2360 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.853+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40300689e0 con 0x7f40300d2360 2024-04-15T15:22:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.853+0000 7f4026ffd640 1 -- 172.21.15.12:0/2803791236 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f402c00cb40 con 0x7f40300d2360 2024-04-15T15:22:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.854+0000 7f4026ffd640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f401406c8f0 0x7f401406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.854+0000 7f4026ffd640 1 -- 172.21.15.12:0/2803791236 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f402c08c360 con 0x7f40300d2360 2024-04-15T15:22:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.854+0000 7f4035cdb640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f401406c8f0 0x7f401406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.855+0000 7f4035cdb640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f401406c8f0 0x7f401406edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f40300cad00 tx=0x7f4020009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:04.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:04.858+0000 7f4026ffd640 1 -- 172.21.15.12:0/2803791236 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f402c017020 con 0x7f40300d2360 2024-04-15T15:22:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.045+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f40300ba500 con 0x7f40300d2360 2024-04-15T15:22:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.046+0000 7f4026ffd640 1 -- 172.21.15.12:0/2803791236 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f402c059be0 con 0x7f40300d2360 2024-04-15T15:22:05.047 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f401406c8f0 msgr2=0x7f401406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f401406c8f0 0x7f401406edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f40300cad00 tx=0x7f4020009040 comp rx=0 tx=0).stop 2024-04-15T15:22:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40300d2360 msgr2=0x7f403015ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40300d2360 0x7f403015ed50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f402c00e9b0 tx=0x7f402c00ee70 comp rx=0 tx=0).stop 2024-04-15T15:22:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 shutdown_connections 2024-04-15T15:22:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f403015e700 0x7f403014b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f401406c8f0 0x7f401406edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40300d3270 0x7f40300c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 --2- 172.21.15.12:0/2803791236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40300d2360 0x7f403015ed50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 >> 172.21.15.12:0/2803791236 conn(0x7f403008bf60 msgr2=0x7f40300d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 shutdown_connections 2024-04-15T15:22:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.048+0000 7f4038740640 1 -- 172.21.15.12:0/2803791236 wait complete. 2024-04-15T15:22:05.060 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.163+0000 7f8359982640 1 Processor -- start 2024-04-15T15:22:05.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.163+0000 7f8359982640 1 -- start start 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.164+0000 7f8359982640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e540 0x7f835409b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.164+0000 7f8359982640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f835406caf0 0x7f835415e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.164+0000 7f8359982640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83540ccc40 con 0x7f835406e920 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.164+0000 7f8359982640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f83540ccf20 con 0x7f835406caf0 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.164+0000 7f8359982640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f83540cd200 con 0x7f835406e540 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.164+0000 7f83537fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f835406caf0 0x7f835415e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.164+0000 7f83537fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f835406caf0 0x7f835415e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52852/0 (socket says 172.21.15.12:52852) 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.164+0000 7f83537fe640 1 -- 172.21.15.12:0/390672001 learned_addr learned my addr 172.21.15.12:0/390672001 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.165+0000 7f8352ffd640 1 --2- 172.21.15.12:0/390672001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e540 0x7f835409b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.165+0000 7f83537fe640 1 -- 172.21.15.12:0/390672001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e540 msgr2=0x7f835409b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.165+0000 7f83537fe640 1 --2- 172.21.15.12:0/390672001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e540 0x7f835409b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.165+0000 7f83537fe640 1 -- 172.21.15.12:0/390672001 >> v1:172.21.15.55:6789/0 conn(0x7f835406e920 legacy=0x7f83540707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.165+0000 7f83537fe640 1 -- 172.21.15.12:0/390672001 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83540cd4e0 con 0x7f835406caf0 2024-04-15T15:22:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.165+0000 7f8352ffd640 1 --2- 172.21.15.12:0/390672001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e540 0x7f835409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.165+0000 7f83537fe640 1 --2- 172.21.15.12:0/390672001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f835415e0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8344009870 tx=0x7f834402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26dd142c5d3b03e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8351ffb640 1 -- 172.21.15.12:0/390672001 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8344034070 con 0x7f835406caf0 2024-04-15T15:22:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8351ffb640 1 -- 172.21.15.12:0/390672001 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8344030840 con 0x7f835406caf0 2024-04-15T15:22:05.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8351ffb640 1 -- 172.21.15.12:0/390672001 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8344030c10 con 0x7f835406caf0 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8359982640 1 -- 172.21.15.12:0/390672001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 msgr2=0x7f835415e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8359982640 1 --2- 172.21.15.12:0/390672001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f835415e0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8344009870 tx=0x7f834402b980 comp rx=0 tx=0).stop 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8359982640 1 -- 172.21.15.12:0/390672001 shutdown_connections 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8359982640 1 --2- 172.21.15.12:0/390672001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f835415e0c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8359982640 1 --2- 172.21.15.12:0/390672001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e540 0x7f835409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8359982640 1 -- 172.21.15.12:0/390672001 >> 172.21.15.12:0/390672001 conn(0x7f83540b3f40 msgr2=0x7f83540b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8359982640 1 -- 172.21.15.12:0/390672001 shutdown_connections 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.166+0000 7f8359982640 1 -- 172.21.15.12:0/390672001 wait complete. 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.167+0000 7f8359982640 1 Processor -- start 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.168+0000 7f8359982640 1 -- start start 2024-04-15T15:22:05.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.168+0000 7f8359982640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f83541f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.168+0000 7f8359982640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f835406e540 0x7f83540c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.168+0000 7f8359982640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e920 0x7f83541f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.168+0000 7f8359982640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8354071060 con 0x7f835406caf0 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.168+0000 7f8359982640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8354070ee0 con 0x7f835406e540 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.168+0000 7f8359982640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f835409b8f0 con 0x7f835406e920 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.168+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f83541f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f83541f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52862/0 (socket says 172.21.15.12:52862) 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8352ffd640 1 -- 172.21.15.12:0/3076437880 learned_addr learned my addr 172.21.15.12:0/3076437880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8352ffd640 1 -- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e920 msgr2=0x7f83541f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8352ffd640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e920 0x7f83541f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8352ffd640 1 -- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f835406e540 msgr2=0x7f83540c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f83527fc640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f835406e540 0x7f83540c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8352ffd640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f835406e540 0x7f83540c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8352ffd640 1 -- 172.21.15.12:0/3076437880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83540b7820 con 0x7f835406caf0 2024-04-15T15:22:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8352ffd640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f83541f3e10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f834c00cf90 tx=0x7f834c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f834bfff640 1 -- 172.21.15.12:0/3076437880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c01a070 con 0x7f835406caf0 2024-04-15T15:22:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f834bfff640 1 -- 172.21.15.12:0/3076437880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f834c00d840 con 0x7f835406caf0 2024-04-15T15:22:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83540b7b00 con 0x7f835406caf0 2024-04-15T15:22:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f834bfff640 1 -- 172.21.15.12:0/3076437880 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c00dc10 con 0x7f835406caf0 2024-04-15T15:22:05.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.169+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83540b80c0 con 0x7f835406caf0 2024-04-15T15:22:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.170+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83540690f0 con 0x7f835406caf0 2024-04-15T15:22:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.171+0000 7f834bfff640 1 -- 172.21.15.12:0/3076437880 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f834c01d070 con 0x7f835406caf0 2024-04-15T15:22:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.171+0000 7f834bfff640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f833406c8f0 0x7f833406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.171+0000 7f834bfff640 1 -- 172.21.15.12:0/3076437880 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f834c08c760 con 0x7f835406caf0 2024-04-15T15:22:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.172+0000 7f83527fc640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f833406c8f0 0x7f833406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.172+0000 7f83527fc640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f833406c8f0 0x7f833406edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f83540ce910 tx=0x7f833c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.176+0000 7f834bfff640 1 -- 172.21.15.12:0/3076437880 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f834c05a100 con 0x7f835406caf0 2024-04-15T15:22:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.374+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f83540d2000 con 0x7f835406caf0 2024-04-15T15:22:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.375+0000 7f834bfff640 1 -- 172.21.15.12:0/3076437880 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f834c05e400 con 0x7f835406caf0 2024-04-15T15:22:05.376 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:05.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f833406c8f0 msgr2=0x7f833406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f833406c8f0 0x7f833406edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f83540ce910 tx=0x7f833c008040 comp rx=0 tx=0).stop 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 msgr2=0x7f83541f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f83541f3e10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f834c00cf90 tx=0x7f834c00a7d0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 shutdown_connections 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f835406e920 0x7f83541f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f833406c8f0 0x7f833406edb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f835406e540 0x7f83540c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 --2- 172.21.15.12:0/3076437880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f835406caf0 0x7f83541f3e10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 >> 172.21.15.12:0/3076437880 conn(0x7f83540b3f40 msgr2=0x7f83541e6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 shutdown_connections 2024-04-15T15:22:05.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.377+0000 7f8359982640 1 -- 172.21.15.12:0/3076437880 wait complete. 2024-04-15T15:22:05.389 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-04-15T15:22:05.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:05.389 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:22:05.418 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:05.418 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:22:05.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:05.418 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:22:05.474 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:22:05.474 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:05.494 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:22:05.494 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:05.553 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.657+0000 7fe928614640 1 Processor -- start 2024-04-15T15:22:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.657+0000 7fe928614640 1 -- start start 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe928614640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe920072d80 0x7fe920071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe928614640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9200af3e0 0x7fe9200af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe928614640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe92015db40 con 0x7fe9200d0690 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe928614640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe92015cfc0 con 0x7fe920072d80 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe928614640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe92015d2a0 con 0x7fe9200af3e0 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe926bb1640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe9200d0690 0x7fe92006e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34682/0 (socket says 172.21.15.12:34682) 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe926bb1640 1 -- 172.21.15.12:0/3836130136 learned_addr learned my addr 172.21.15.12:0/3836130136 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9263b0640 1 --2- 172.21.15.12:0/3836130136 >> v2:172.21.15.55:3300/0 conn(0x7fe920072d80 0x7fe920071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe925baf640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9200af3e0 0x7fe9200af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3557118789 0 0) 0x7fe92015db40 con 0x7fe9200d0690 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe90c003520 con 0x7fe9200d0690 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2830498081 0 0) 0x7fe90c003520 con 0x7fe9200d0690 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9200af3e0 msgr2=0x7fe9200af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9200af3e0 0x7fe9200af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 >> v2:172.21.15.55:3300/0 conn(0x7fe920072d80 msgr2=0x7fe920071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 --2- 172.21.15.12:0/3836130136 >> v2:172.21.15.55:3300/0 conn(0x7fe920072d80 0x7fe920071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92015d580 con 0x7fe9200d0690 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fe9180019f0 con 0x7fe9200d0690 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 >> v1:172.21.15.12:6789/0 conn(0x7fe9200d0690 legacy=0x7fe92006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe90c0039b0 0x7fe90c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c004800 0x7fe90c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9200af3e0 0x7fe90c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe92015cfc0 con 0x7fe90c0039b0 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe92015d2a0 con 0x7fe90c004800 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.658+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe90c00b9c0 con 0x7fe9200af3e0 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe9263b0640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe90c0039b0 0x7fe90c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe926bb1640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9200af3e0 0x7fe90c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe925baf640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c004800 0x7fe90c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe9263b0640 1 -- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9200af3e0 msgr2=0x7fe90c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe9263b0640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9200af3e0 0x7fe90c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe9263b0640 1 -- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c004800 msgr2=0x7fe90c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe9263b0640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c004800 0x7fe90c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe9263b0640 1 -- 172.21.15.12:0/3836130136 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c003520 con 0x7fe90c0039b0 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.659+0000 7fe9263b0640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe90c0039b0 0x7fe90c003dd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009af0 tx=0x7fe90c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0fd76f37d5c6f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c039070 con 0x7fe90c0039b0 2024-04-15T15:22:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe91c02c710 con 0x7fe90c0039b0 2024-04-15T15:22:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe9253ae640 1 -- 172.21.15.12:0/3836130136 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c009690 con 0x7fe90c0039b0 2024-04-15T15:22:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe928614640 1 -- 172.21.15.12:0/3836130136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe90c0039b0 msgr2=0x7fe90c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe928614640 1 --2- 172.21.15.12:0/3836130136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe90c0039b0 0x7fe90c003dd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009af0 tx=0x7fe90c005830 comp rx=0 tx=0).stop 2024-04-15T15:22:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe926bb1640 1 -- 172.21.15.12:0/3836130136 reap_dead start 2024-04-15T15:22:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe928614640 1 -- 172.21.15.12:0/3836130136 shutdown_connections 2024-04-15T15:22:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe928614640 1 -- 172.21.15.12:0/3836130136 >> 172.21.15.12:0/3836130136 conn(0x7fe92006be00 msgr2=0x7fe92006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe928614640 1 -- 172.21.15.12:0/3836130136 shutdown_connections 2024-04-15T15:22:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.660+0000 7fe928614640 1 -- 172.21.15.12:0/3836130136 wait complete. 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.661+0000 7fe928614640 1 Processor -- start 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe928614640 1 -- start start 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe928614640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe90c0039b0 0x7fe92014c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe928614640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9200cf7b0 0x7fe920156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe928614640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe92009a5d0 0x7fe92009aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe928614640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe92015cd10 con 0x7fe92009a5d0 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe928614640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe92015cb90 con 0x7fe9200cf7b0 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe928614640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe9200af840 con 0x7fe90c0039b0 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe926bb1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe92009a5d0 0x7fe92009aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe926bb1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe92009a5d0 0x7fe92009aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52884/0 (socket says 172.21.15.12:52884) 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.662+0000 7fe926bb1640 1 -- 172.21.15.12:0/225742144 learned_addr learned my addr 172.21.15.12:0/225742144 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:05.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe9263b0640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe90c0039b0 0x7fe92014c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe925baf640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9200cf7b0 0x7fe920156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe926bb1640 1 -- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe90c0039b0 msgr2=0x7fe92014c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe926bb1640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe90c0039b0 0x7fe92014c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe926bb1640 1 -- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9200cf7b0 msgr2=0x7fe920156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe926bb1640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9200cf7b0 0x7fe920156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe926bb1640 1 -- 172.21.15.12:0/225742144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe920156f60 con 0x7fe92009a5d0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe926bb1640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe92009a5d0 0x7fe92009aa50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe9200736a0 tx=0x7fe91800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe916ffd640 1 -- 172.21.15.12:0/225742144 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91800ec40 con 0x7fe92009a5d0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe916ffd640 1 -- 172.21.15.12:0/225742144 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe91800ede0 con 0x7fe92009a5d0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe920157240 con 0x7fe92009a5d0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe916ffd640 1 -- 172.21.15.12:0/225742144 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe918014670 con 0x7fe92009a5d0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.663+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe920157800 con 0x7fe92009a5d0 2024-04-15T15:22:05.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.664+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9200689e0 con 0x7fe92009a5d0 2024-04-15T15:22:05.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.665+0000 7fe916ffd640 1 -- 172.21.15.12:0/225742144 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe918014810 con 0x7fe92009a5d0 2024-04-15T15:22:05.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.665+0000 7fe916ffd640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe90406c8f0 0x7fe90406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.665+0000 7fe916ffd640 1 -- 172.21.15.12:0/225742144 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fe91808be10 con 0x7fe92009a5d0 2024-04-15T15:22:05.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.665+0000 7fe9263b0640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe90406c8f0 0x7fe90406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.666+0000 7fe9263b0640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe90406c8f0 0x7fe90406edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe91c036000 tx=0x7fe91c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.670+0000 7fe916ffd640 1 -- 172.21.15.12:0/225742144 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe918059d70 con 0x7fe92009a5d0 2024-04-15T15:22:05.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.854+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe920073210 con 0x7fe92009a5d0 2024-04-15T15:22:05.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.855+0000 7fe916ffd640 1 -- 172.21.15.12:0/225742144 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7fe918059670 con 0x7fe92009a5d0 2024-04-15T15:22:05.857 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:05.857 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:05.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.857+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe90406c8f0 msgr2=0x7fe90406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.857+0000 7fe928614640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe90406c8f0 0x7fe90406edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe91c036000 tx=0x7fe91c030000 comp rx=0 tx=0).stop 2024-04-15T15:22:05.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.857+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe92009a5d0 msgr2=0x7fe92009aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.857+0000 7fe928614640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe92009a5d0 0x7fe92009aa50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe9200736a0 tx=0x7fe91800e530 comp rx=0 tx=0).stop 2024-04-15T15:22:05.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.858+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 shutdown_connections 2024-04-15T15:22:05.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.858+0000 7fe928614640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe92009a5d0 0x7fe92009aa50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.858+0000 7fe928614640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe90406c8f0 0x7fe90406edb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.858+0000 7fe928614640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9200cf7b0 0x7fe920156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.858+0000 7fe928614640 1 --2- 172.21.15.12:0/225742144 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe90c0039b0 0x7fe92014c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.858+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 >> 172.21.15.12:0/225742144 conn(0x7fe92006be00 msgr2=0x7fe9200b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:05.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.858+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 shutdown_connections 2024-04-15T15:22:05.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.858+0000 7fe928614640 1 -- 172.21.15.12:0/225742144 wait complete. 2024-04-15T15:22:05.870 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:05.871 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:05.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.978+0000 7fd7cc8ba640 1 Processor -- start 2024-04-15T15:22:05.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.978+0000 7fd7cc8ba640 1 -- start start 2024-04-15T15:22:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7cc8ba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd7c80b3160 0x7fd7c815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7cc8ba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80ae280 0x7fd7c80ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7cc8ba640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c815cb50 con 0x7fd7c80d0690 2024-04-15T15:22:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7cc8ba640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c815bfd0 con 0x7fd7c80b3160 2024-04-15T15:22:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7cc8ba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c815c2b0 con 0x7fd7c80ae280 2024-04-15T15:22:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7c659c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd7c80d0690 0x7fd7c80d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34686/0 (socket says 172.21.15.12:34686) 2024-04-15T15:22:05.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7c659c640 1 -- 172.21.15.12:0/4227968477 learned_addr learned my addr 172.21.15.12:0/4227968477 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7c5d9b640 1 --2- 172.21.15.12:0/4227968477 >> v2:172.21.15.55:3300/0 conn(0x7fd7c80b3160 0x7fd7c815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7c6d9d640 1 --2- 172.21.15.12:0/4227968477 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80ae280 0x7fd7c80ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7c559a640 1 -- 172.21.15.12:0/4227968477 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1325957061 0 0) 0x7fd7c815cb50 con 0x7fd7c80d0690 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.979+0000 7fd7c559a640 1 -- 172.21.15.12:0/4227968477 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b4003520 con 0x7fd7c80d0690 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.980+0000 7fd7c559a640 1 -- 172.21.15.12:0/4227968477 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3205480021 0 0) 0x7fd7b4003520 con 0x7fd7c80d0690 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.980+0000 7fd7c5d9b640 1 -- 172.21.15.12:0/4227968477 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80ae280 msgr2=0x7fd7c80ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.980+0000 7fd7c5d9b640 1 --2- 172.21.15.12:0/4227968477 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80ae280 0x7fd7c80ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.980+0000 7fd7c5d9b640 1 -- 172.21.15.12:0/4227968477 >> v1:172.21.15.12:6789/0 conn(0x7fd7c80d0690 legacy=0x7fd7c80d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.980+0000 7fd7c5d9b640 1 -- 172.21.15.12:0/4227968477 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c815c590 con 0x7fd7c80b3160 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.980+0000 7fd7c5d9b640 1 --2- 172.21.15.12:0/4227968477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80b3160 0x7fd7c815b690 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd7b00097a0 tx=0x7fd7b002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f451f01c8da547d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7c559a640 1 -- 172.21.15.12:0/4227968477 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0037070 con 0x7fd7c80b3160 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7c559a640 1 -- 172.21.15.12:0/4227968477 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b0030510 con 0x7fd7c80b3160 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7c559a640 1 -- 172.21.15.12:0/4227968477 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00019f0 con 0x7fd7c80b3160 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/4227968477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80b3160 msgr2=0x7fd7c815b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/4227968477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80b3160 0x7fd7c815b690 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd7b00097a0 tx=0x7fd7b002b790 comp rx=0 tx=0).stop 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/4227968477 shutdown_connections 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/4227968477 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80ae280 0x7fd7c80ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/4227968477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80b3160 0x7fd7c815b690 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/4227968477 >> 172.21.15.12:0/4227968477 conn(0x7fd7c806be00 msgr2=0x7fd7c806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/4227968477 shutdown_connections 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.981+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/4227968477 wait complete. 2024-04-15T15:22:05.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7cc8ba640 1 Processor -- start 2024-04-15T15:22:05.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7cc8ba640 1 -- start start 2024-04-15T15:22:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7cc8ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80ae280 0x7fd7c81574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7cc8ba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7c80b3160 0x7fd7c81522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7cc8ba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80d0690 0x7fd7c806dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7cc8ba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c815bbd0 con 0x7fd7c80b3160 2024-04-15T15:22:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7cc8ba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c80b4fb0 con 0x7fd7c80ae280 2024-04-15T15:22:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7cc8ba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c815bd50 con 0x7fd7c80d0690 2024-04-15T15:22:05.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7c5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7c80b3160 0x7fd7c81522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7c5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7c80b3160 0x7fd7c81522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52900/0 (socket says 172.21.15.12:52900) 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.983+0000 7fd7c5d9b640 1 -- 172.21.15.12:0/2586308574 learned_addr learned my addr 172.21.15.12:0/2586308574 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c659c640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80ae280 0x7fd7c81574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c6d9d640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80d0690 0x7fd7c806dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c5d9b640 1 -- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80d0690 msgr2=0x7fd7c806dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c5d9b640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80d0690 0x7fd7c806dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c5d9b640 1 -- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80ae280 msgr2=0x7fd7c81574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c5d9b640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80ae280 0x7fd7c81574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c5d9b640 1 -- 172.21.15.12:0/2586308574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c80cf860 con 0x7fd7c80b3160 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c659c640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80ae280 0x7fd7c81574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c6d9d640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80d0690 0x7fd7c806dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7c5d9b640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7c80b3160 0x7fd7c81522c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd7b002b7d0 tx=0x7fd7b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7beffd640 1 -- 172.21.15.12:0/2586308574 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0041020 con 0x7fd7c80b3160 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7beffd640 1 -- 172.21.15.12:0/2586308574 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b003c870 con 0x7fd7c80b3160 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c815f300 con 0x7fd7c80b3160 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7beffd640 1 -- 172.21.15.12:0/2586308574 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0037040 con 0x7fd7c80b3160 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.984+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c815f8c0 con 0x7fd7c80b3160 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.985+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c80689e0 con 0x7fd7c80b3160 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.985+0000 7fd7beffd640 1 -- 172.21.15.12:0/2586308574 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7b0044020 con 0x7fd7c80b3160 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.986+0000 7fd7beffd640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd7a806c8f0 0x7fd7a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.986+0000 7fd7beffd640 1 -- 172.21.15.12:0/2586308574 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fd7b00ad9d0 con 0x7fd7c80b3160 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.986+0000 7fd7c659c640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd7a806c8f0 0x7fd7a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:05.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.987+0000 7fd7c659c640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd7a806c8f0 0x7fd7a806edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd7c000e270 tx=0x7fd7c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:05.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:05.990+0000 7fd7beffd640 1 -- 172.21.15.12:0/2586308574 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7b004b030 con 0x7fd7c80b3160 2024-04-15T15:22:06.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.173+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd7c806fae0 con 0x7fd7c80b3160 2024-04-15T15:22:06.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.174+0000 7fd7beffd640 1 -- 172.21.15.12:0/2586308574 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fd7b007b9d0 con 0x7fd7c80b3160 2024-04-15T15:22:06.175 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd7a806c8f0 msgr2=0x7fd7a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd7a806c8f0 0x7fd7a806edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd7c000e270 tx=0x7fd7c0009040 comp rx=0 tx=0).stop 2024-04-15T15:22:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7c80b3160 msgr2=0x7fd7c81522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7c80b3160 0x7fd7c81522c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd7b002b7d0 tx=0x7fd7b00046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 shutdown_connections 2024-04-15T15:22:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7c80d0690 0x7fd7c806dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7c80b3160 0x7fd7c81522c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd7a806c8f0 0x7fd7a806edb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 --2- 172.21.15.12:0/2586308574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7c80ae280 0x7fd7c81574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 >> 172.21.15.12:0/2586308574 conn(0x7fd7c806be00 msgr2=0x7fd7c80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 shutdown_connections 2024-04-15T15:22:06.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.176+0000 7fd7cc8ba640 1 -- 172.21.15.12:0/2586308574 wait complete. 2024-04-15T15:22:06.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:06.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.294+0000 7f56d0893640 1 Processor -- start 2024-04-15T15:22:06.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.294+0000 7f56d0893640 1 -- start start 2024-04-15T15:22:06.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56d0893640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56cc09bb50 0x7f56cc167600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56d0893640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56cc15dbd0 con 0x7f56cc0cd000 2024-04-15T15:22:06.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56d0893640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56cc0cd6a0 con 0x7f56cc1602d0 2024-04-15T15:22:06.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56d0893640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f56cc0cd460 con 0x7f56cc09bb50 2024-04-15T15:22:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56cad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f56cc09bb50 0x7f56cc167600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56c1d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f56cc0cd000 0x7f56cc0d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34700/0 (socket says 172.21.15.12:34700) 2024-04-15T15:22:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56c1d9b640 1 -- 172.21.15.12:0/2308963713 learned_addr learned my addr 172.21.15.12:0/2308963713 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56cad9d640 1 -- 172.21.15.12:0/2308963713 >> v1:172.21.15.55:6789/0 conn(0x7f56cc1602d0 legacy=0x7f56cc1606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56cad9d640 1 -- 172.21.15.12:0/2308963713 >> v1:172.21.15.12:6789/0 conn(0x7f56cc0cd000 legacy=0x7f56cc0d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56cad9d640 1 -- 172.21.15.12:0/2308963713 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56cc0cd980 con 0x7f56cc09bb50 2024-04-15T15:22:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.295+0000 7f56c9d9b640 1 -- 172.21.15.12:0/2308963713 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2842270375 0 0) 0x7f56cc15dbd0 con 0x7f56cc0cd000 2024-04-15T15:22:06.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56cad9d640 1 --2- 172.21.15.12:0/2308963713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc167600 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009d90 tx=0x7f56c402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821acffdd43128c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56c9d9b640 1 -- 172.21.15.12:0/2308963713 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c4039070 con 0x7f56cc09bb50 2024-04-15T15:22:06.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56c9d9b640 1 -- 172.21.15.12:0/2308963713 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56c402cd00 con 0x7f56cc09bb50 2024-04-15T15:22:06.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56c9d9b640 1 -- 172.21.15.12:0/2308963713 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c4034660 con 0x7f56cc09bb50 2024-04-15T15:22:06.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56d0893640 1 -- 172.21.15.12:0/2308963713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 msgr2=0x7f56cc167600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56d0893640 1 --2- 172.21.15.12:0/2308963713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc167600 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009d90 tx=0x7f56c402bf50 comp rx=0 tx=0).stop 2024-04-15T15:22:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56d0893640 1 -- 172.21.15.12:0/2308963713 shutdown_connections 2024-04-15T15:22:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56d0893640 1 --2- 172.21.15.12:0/2308963713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc167600 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.296+0000 7f56d0893640 1 -- 172.21.15.12:0/2308963713 >> 172.21.15.12:0/2308963713 conn(0x7f56cc0d2460 msgr2=0x7f56cc0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.297+0000 7f56d0893640 1 -- 172.21.15.12:0/2308963713 shutdown_connections 2024-04-15T15:22:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.297+0000 7f56d0893640 1 -- 172.21.15.12:0/2308963713 wait complete. 2024-04-15T15:22:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.298+0000 7f56d0893640 1 Processor -- start 2024-04-15T15:22:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.298+0000 7f56d0893640 1 -- start start 2024-04-15T15:22:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56d0893640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc0c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56d0893640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f56cc0cd000 0x7f56cc154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56d0893640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56cc1602d0 0x7f56cc155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56d0893640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc0ce220 con 0x7f56cc09bb50 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56d0893640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc0cdf40 con 0x7f56cc1602d0 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56d0893640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f56cc167b40 con 0x7f56cc0cd000 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc0c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc0c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52918/0 (socket says 172.21.15.12:52918) 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 -- 172.21.15.12:0/2166312741 learned_addr learned my addr 172.21.15.12:0/2166312741 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56c1d9b640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f56cc0cd000 0x7f56cc154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56cad9d640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56cc1602d0 0x7f56cc155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 -- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f56cc0cd000 msgr2=0x7f56cc154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f56cc0cd000 0x7f56cc154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 -- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56cc1602d0 msgr2=0x7f56cc155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56cc1602d0 0x7f56cc155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 -- 172.21.15.12:0/2166312741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56cc0c7860 con 0x7f56cc09bb50 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.299+0000 7f56ca59c640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc0c8680 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00c430 tx=0x7f56bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.300+0000 7f56c37fe640 1 -- 172.21.15.12:0/2166312741 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc017070 con 0x7f56cc09bb50 2024-04-15T15:22:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.300+0000 7f56c37fe640 1 -- 172.21.15.12:0/2166312741 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56bc0052d0 con 0x7f56cc09bb50 2024-04-15T15:22:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.300+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56cc0c7b40 con 0x7f56cc09bb50 2024-04-15T15:22:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.300+0000 7f56c37fe640 1 -- 172.21.15.12:0/2166312741 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc003830 con 0x7f56cc09bb50 2024-04-15T15:22:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.300+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56cc0c8bc0 con 0x7f56cc09bb50 2024-04-15T15:22:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.301+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56cc0689e0 con 0x7f56cc09bb50 2024-04-15T15:22:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.301+0000 7f56c37fe640 1 -- 172.21.15.12:0/2166312741 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56bc01a070 con 0x7f56cc09bb50 2024-04-15T15:22:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.302+0000 7f56c37fe640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f56ac06c8f0 0x7f56ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.302+0000 7f56c37fe640 1 -- 172.21.15.12:0/2166312741 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f56bc08b1d0 con 0x7f56cc09bb50 2024-04-15T15:22:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.302+0000 7f56c1d9b640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f56ac06c8f0 0x7f56ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.303+0000 7f56c1d9b640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f56ac06c8f0 0x7f56ac06edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f56b4007f70 tx=0x7f56b4007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.306+0000 7f56c37fe640 1 -- 172.21.15.12:0/2166312741 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f56bc058c00 con 0x7f56cc09bb50 2024-04-15T15:22:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.491+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f56cc0cec50 con 0x7f56cc09bb50 2024-04-15T15:22:06.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.491+0000 7f56c37fe640 1 -- 172.21.15.12:0/2166312741 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f56bc05cf00 con 0x7f56cc09bb50 2024-04-15T15:22:06.493 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f56ac06c8f0 msgr2=0x7f56ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f56ac06c8f0 0x7f56ac06edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f56b4007f70 tx=0x7f56b4007f00 comp rx=0 tx=0).stop 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 msgr2=0x7f56cc0c8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc0c8680 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00c430 tx=0x7f56bc00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 shutdown_connections 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f56ac06c8f0 0x7f56ac06edb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56cc1602d0 0x7f56cc155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f56cc0cd000 0x7f56cc154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 --2- 172.21.15.12:0/2166312741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56cc09bb50 0x7f56cc0c8680 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 >> 172.21.15.12:0/2166312741 conn(0x7f56cc0d2460 msgr2=0x7f56cc071720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:06.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 shutdown_connections 2024-04-15T15:22:06.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.493+0000 7f56d0893640 1 -- 172.21.15.12:0/2166312741 wait complete. 2024-04-15T15:22:06.506 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.612+0000 7f15236e4640 1 Processor -- start 2024-04-15T15:22:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.612+0000 7f15236e4640 1 -- start start 2024-04-15T15:22:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.613+0000 7f15236e4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f151c0701f0 0x7f151c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.613+0000 7f15236e4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f151c15fb30 0x7f151c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.613+0000 7f15236e4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f151c070720 con 0x7f151c09a9d0 2024-04-15T15:22:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.613+0000 7f15236e4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f151c09ba90 con 0x7f151c0701f0 2024-04-15T15:22:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.613+0000 7f15236e4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f151c0d08c0 con 0x7f151c15fb30 2024-04-15T15:22:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.613+0000 7f1520c7f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f151c0701f0 0x7f151c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.613+0000 7f1521480640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f151c09a9d0 0x7f151c09adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34716/0 (socket says 172.21.15.12:34716) 2024-04-15T15:22:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.613+0000 7f1521480640 1 -- 172.21.15.12:0/1131280936 learned_addr learned my addr 172.21.15.12:0/1131280936 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.614+0000 7f1521c81640 1 --2- 172.21.15.12:0/1131280936 >> v2:172.21.15.55:3300/0 conn(0x7f151c15fb30 0x7f151c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.614+0000 7f1520c7f640 1 -- 172.21.15.12:0/1131280936 >> v2:172.21.15.55:3300/0 conn(0x7f151c15fb30 msgr2=0x7f151c0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.614+0000 7f1520c7f640 1 --2- 172.21.15.12:0/1131280936 >> v2:172.21.15.55:3300/0 conn(0x7f151c15fb30 0x7f151c0d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.614+0000 7f1520c7f640 1 -- 172.21.15.12:0/1131280936 >> v1:172.21.15.12:6789/0 conn(0x7f151c09a9d0 legacy=0x7f151c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.614+0000 7f1513fff640 1 -- 172.21.15.12:0/1131280936 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635935948 0 0) 0x7f151c070720 con 0x7f151c09a9d0 2024-04-15T15:22:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.614+0000 7f1520c7f640 1 -- 172.21.15.12:0/1131280936 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151c070a90 con 0x7f151c0701f0 2024-04-15T15:22:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.614+0000 7f1521c81640 1 --2- 172.21.15.12:0/1131280936 >> v2:172.21.15.55:3300/0 conn(0x7f151c15fb30 0x7f151c0d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.614+0000 7f1520c7f640 1 --2- 172.21.15.12:0/1131280936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c0701f0 0x7f151c160420 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f151c073740 tx=0x7f150c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289a4afbb0f90c25 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f1513fff640 1 -- 172.21.15.12:0/1131280936 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c035020 con 0x7f151c0701f0 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f1513fff640 1 -- 172.21.15.12:0/1131280936 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f150c031830 con 0x7f151c0701f0 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f1513fff640 1 -- 172.21.15.12:0/1131280936 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c031c00 con 0x7f151c0701f0 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f15236e4640 1 -- 172.21.15.12:0/1131280936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c0701f0 msgr2=0x7f151c160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f15236e4640 1 --2- 172.21.15.12:0/1131280936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c0701f0 0x7f151c160420 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f151c073740 tx=0x7f150c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f15236e4640 1 -- 172.21.15.12:0/1131280936 shutdown_connections 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f15236e4640 1 --2- 172.21.15.12:0/1131280936 >> v2:172.21.15.55:3300/0 conn(0x7f151c15fb30 0x7f151c0d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f15236e4640 1 --2- 172.21.15.12:0/1131280936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c0701f0 0x7f151c160420 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f15236e4640 1 -- 172.21.15.12:0/1131280936 >> 172.21.15.12:0/1131280936 conn(0x7f151c0d2460 msgr2=0x7f151c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f15236e4640 1 -- 172.21.15.12:0/1131280936 shutdown_connections 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.615+0000 7f15236e4640 1 -- 172.21.15.12:0/1131280936 wait complete. 2024-04-15T15:22:06.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.616+0000 7f15236e4640 1 Processor -- start 2024-04-15T15:22:06.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.616+0000 7f15236e4640 1 -- start start 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f15236e4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f151c0701f0 0x7f151c15b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f15236e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c09a9d0 0x7f151c14da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f15236e4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f151c15fb30 0x7f151c0c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f15236e4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f151c15ffc0 con 0x7f151c09a9d0 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f15236e4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f151c071050 con 0x7f151c15fb30 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f15236e4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f151c0b1c60 con 0x7f151c0701f0 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f1520c7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c09a9d0 0x7f151c14da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f1520c7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c09a9d0 0x7f151c14da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52926/0 (socket says 172.21.15.12:52926) 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f1520c7f640 1 -- 172.21.15.12:0/2027736824 learned_addr learned my addr 172.21.15.12:0/2027736824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f1521c81640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f151c15fb30 0x7f151c0c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.617+0000 7f1521480640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f151c0701f0 0x7f151c15b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1520c7f640 1 -- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f151c0701f0 msgr2=0x7f151c15b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1520c7f640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f151c0701f0 0x7f151c15b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1520c7f640 1 -- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f151c15fb30 msgr2=0x7f151c0c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1520c7f640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f151c15fb30 0x7f151c0c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1520c7f640 1 -- 172.21.15.12:0/2027736824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151c0c72b0 con 0x7f151c09a9d0 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1520c7f640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c09a9d0 0x7f151c14da40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f150c034040 tx=0x7f150c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1511ffb640 1 -- 172.21.15.12:0/2027736824 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c041070 con 0x7f151c09a9d0 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1511ffb640 1 -- 172.21.15.12:0/2027736824 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f150c004230 con 0x7f151c09a9d0 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151c0c7590 con 0x7f151c09a9d0 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f1511ffb640 1 -- 172.21.15.12:0/2027736824 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c035040 con 0x7f151c09a9d0 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.618+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151c0b5570 con 0x7f151c09a9d0 2024-04-15T15:22:06.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.619+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f151c0689e0 con 0x7f151c09a9d0 2024-04-15T15:22:06.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.620+0000 7f1511ffb640 1 -- 172.21.15.12:0/2027736824 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f150c038020 con 0x7f151c09a9d0 2024-04-15T15:22:06.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.620+0000 7f1511ffb640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f150006c8f0 0x7f150006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.620+0000 7f1511ffb640 1 -- 172.21.15.12:0/2027736824 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f150c0ad8f0 con 0x7f151c09a9d0 2024-04-15T15:22:06.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.621+0000 7f1521480640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f150006c8f0 0x7f150006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.621+0000 7f1521480640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f150006c8f0 0x7f150006edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1514000f60 tx=0x7f1514007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.625+0000 7f1511ffb640 1 -- 172.21.15.12:0/2027736824 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f150c07b270 con 0x7f151c09a9d0 2024-04-15T15:22:06.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.812+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f151c070720 con 0x7f151c09a9d0 2024-04-15T15:22:06.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.812+0000 7f1511ffb640 1 -- 172.21.15.12:0/2027736824 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f150c07f570 con 0x7f151c09a9d0 2024-04-15T15:22:06.814 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f150006c8f0 msgr2=0x7f150006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f150006c8f0 0x7f150006edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1514000f60 tx=0x7f1514007d50 comp rx=0 tx=0).stop 2024-04-15T15:22:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c09a9d0 msgr2=0x7f151c14da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c09a9d0 0x7f151c14da40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f150c034040 tx=0x7f150c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 shutdown_connections 2024-04-15T15:22:06.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f150006c8f0 0x7f150006edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f151c15fb30 0x7f151c0c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f151c09a9d0 0x7f151c14da40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 --2- 172.21.15.12:0/2027736824 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f151c0701f0 0x7f151c15b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.814+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 >> 172.21.15.12:0/2027736824 conn(0x7f151c0d2460 msgr2=0x7f151c0b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.815+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 shutdown_connections 2024-04-15T15:22:06.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.815+0000 7f15236e4640 1 -- 172.21.15.12:0/2027736824 wait complete. 2024-04-15T15:22:06.826 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:06.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.932+0000 7f49c0460640 1 Processor -- start 2024-04-15T15:22:06.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.932+0000 7f49c0460640 1 -- start start 2024-04-15T15:22:06.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49c0460640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f49b815e9a0 0x7f49b80d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49c0460640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f49b80d1400 0x7f49b80cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49c0460640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b80d1e60 con 0x7f49b80cf750 2024-04-15T15:22:06.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49c0460640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f49b80d18a0 con 0x7f49b80d1400 2024-04-15T15:22:06.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49c0460640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f49b80d1b80 con 0x7f49b815e9a0 2024-04-15T15:22:06.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49bd9fb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f49b80d1400 0x7f49b80cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49be9fd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f49b80cf750 0x7f49b80d2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34724/0 (socket says 172.21.15.12:34724) 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49bd9fb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f49b80d1400 0x7f49b80cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52940/0 (socket says 172.21.15.12:52940) 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49be9fd640 1 -- 172.21.15.12:0/1464835152 learned_addr learned my addr 172.21.15.12:0/1464835152 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49bd9fb640 1 -- 172.21.15.12:0/1464835152 >> v2:172.21.15.55:3300/0 conn(0x7f49b815e9a0 msgr2=0x7f49b80d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.933+0000 7f49be1fc640 1 --2- 172.21.15.12:0/1464835152 >> v2:172.21.15.55:3300/0 conn(0x7f49b815e9a0 0x7f49b80d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.934+0000 7f49bd9fb640 1 --2- 172.21.15.12:0/1464835152 >> v2:172.21.15.55:3300/0 conn(0x7f49b815e9a0 0x7f49b80d0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.934+0000 7f49bd9fb640 1 -- 172.21.15.12:0/1464835152 >> v1:172.21.15.12:6789/0 conn(0x7f49b80cf750 legacy=0x7f49b80d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.934+0000 7f49bd9fb640 1 -- 172.21.15.12:0/1464835152 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b806cff0 con 0x7f49b80d1400 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.934+0000 7f49bd1fa640 1 -- 172.21.15.12:0/1464835152 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506633506 0 0) 0x7f49b80d1e60 con 0x7f49b80cf750 2024-04-15T15:22:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.934+0000 7f49bd9fb640 1 --2- 172.21.15.12:0/1464835152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80d1400 0x7f49b80cf210 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f49b808dfc0 tx=0x7f49a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e029fd3748ba0ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.937+0000 7f49bd1fa640 1 -- 172.21.15.12:0/1464835152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8035020 con 0x7f49b80d1400 2024-04-15T15:22:06.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.937+0000 7f49bd1fa640 1 -- 172.21.15.12:0/1464835152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a8031830 con 0x7f49b80d1400 2024-04-15T15:22:06.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.937+0000 7f49bd1fa640 1 -- 172.21.15.12:0/1464835152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8031c00 con 0x7f49b80d1400 2024-04-15T15:22:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.938+0000 7f49c0460640 1 -- 172.21.15.12:0/1464835152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80d1400 msgr2=0x7f49b80cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.938+0000 7f49c0460640 1 --2- 172.21.15.12:0/1464835152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80d1400 0x7f49b80cf210 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f49b808dfc0 tx=0x7f49a802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.938+0000 7f49c0460640 1 -- 172.21.15.12:0/1464835152 shutdown_connections 2024-04-15T15:22:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.938+0000 7f49c0460640 1 --2- 172.21.15.12:0/1464835152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80d1400 0x7f49b80cf210 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.938+0000 7f49c0460640 1 --2- 172.21.15.12:0/1464835152 >> v2:172.21.15.55:3300/0 conn(0x7f49b815e9a0 0x7f49b80d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.938+0000 7f49c0460640 1 -- 172.21.15.12:0/1464835152 >> 172.21.15.12:0/1464835152 conn(0x7f49b809d950 msgr2=0x7f49b808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:06.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.938+0000 7f49c0460640 1 -- 172.21.15.12:0/1464835152 shutdown_connections 2024-04-15T15:22:06.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.938+0000 7f49c0460640 1 -- 172.21.15.12:0/1464835152 wait complete. 2024-04-15T15:22:06.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.939+0000 7f49c0460640 1 Processor -- start 2024-04-15T15:22:06.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.939+0000 7f49c0460640 1 -- start start 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49c0460640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80cf750 0x7f49b81f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49c0460640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80d1400 0x7f49b8157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49c0460640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f49b815e9a0 0x7f49b81f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49c0460640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f49b80d24e0 con 0x7f49b80cf750 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49c0460640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f49b80d2360 con 0x7f49b80d1400 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49c0460640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f49b80d32b0 con 0x7f49b815e9a0 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80cf750 0x7f49b81f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80cf750 0x7f49b81f8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52942/0 (socket says 172.21.15.12:52942) 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 -- 172.21.15.12:0/882821854 learned_addr learned my addr 172.21.15.12:0/882821854 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:06.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49bd9fb640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80d1400 0x7f49b8157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be9fd640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f49b815e9a0 0x7f49b81f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 -- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f49b815e9a0 msgr2=0x7f49b81f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f49b815e9a0 0x7f49b81f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 -- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80d1400 msgr2=0x7f49b8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80d1400 0x7f49b8157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 -- 172.21.15.12:0/882821854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b8160090 con 0x7f49b80cf750 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be9fd640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f49b815e9a0 0x7f49b81f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49be1fc640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80cf750 0x7f49b81f8820 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f49b400c820 tx=0x7f49b400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.940+0000 7f49aeffd640 1 -- 172.21.15.12:0/882821854 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4015070 con 0x7f49b80cf750 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.941+0000 7f49aeffd640 1 -- 172.21.15.12:0/882821854 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b4004020 con 0x7f49b80cf750 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.941+0000 7f49aeffd640 1 -- 172.21.15.12:0/882821854 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b40043f0 con 0x7f49b80cf750 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.941+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b8160370 con 0x7f49b80cf750 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.941+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b80ce2b0 con 0x7f49b80cf750 2024-04-15T15:22:06.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.942+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b80690f0 con 0x7f49b80cf750 2024-04-15T15:22:06.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.944+0000 7f49aeffd640 1 -- 172.21.15.12:0/882821854 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f49b4007830 con 0x7f49b80cf750 2024-04-15T15:22:06.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.944+0000 7f49aeffd640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f499c06c8f0 0x7f499c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:06.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.944+0000 7f49aeffd640 1 -- 172.21.15.12:0/882821854 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f49b408c040 con 0x7f49b80cf750 2024-04-15T15:22:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.944+0000 7f49bd9fb640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f499c06c8f0 0x7f499c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:06.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.945+0000 7f49bd9fb640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f499c06c8f0 0x7f499c06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f49a8036000 tx=0x7f49a8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:06.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:06.948+0000 7f49aeffd640 1 -- 172.21.15.12:0/882821854 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f49b4059b50 con 0x7f49b80cf750 2024-04-15T15:22:07.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.130+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f49b806cd30 con 0x7f49b80cf750 2024-04-15T15:22:07.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.131+0000 7f49aeffd640 1 -- 172.21.15.12:0/882821854 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f49b405de50 con 0x7f49b80cf750 2024-04-15T15:22:07.132 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:07.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f499c06c8f0 msgr2=0x7f499c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f499c06c8f0 0x7f499c06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f49a8036000 tx=0x7f49a8008030 comp rx=0 tx=0).stop 2024-04-15T15:22:07.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80cf750 msgr2=0x7f49b81f8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80cf750 0x7f49b81f8820 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f49b400c820 tx=0x7f49b400cce0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 shutdown_connections 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f49b815e9a0 0x7f49b81f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f499c06c8f0 0x7f499c06edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80d1400 0x7f49b8157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 --2- 172.21.15.12:0/882821854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49b80cf750 0x7f49b81f8820 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 >> 172.21.15.12:0/882821854 conn(0x7f49b809d950 msgr2=0x7f49b815dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 shutdown_connections 2024-04-15T15:22:07.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.133+0000 7f49c0460640 1 -- 172.21.15.12:0/882821854 wait complete. 2024-04-15T15:22:07.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.249+0000 7fc6b358f640 1 Processor -- start 2024-04-15T15:22:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.250+0000 7fc6b358f640 1 -- start start 2024-04-15T15:22:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.250+0000 7fc6b358f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6ac15e790 0x7fc6ac0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.250+0000 7fc6b358f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc6ac0d1cf0 0x7fc6ac0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.250+0000 7fc6b358f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6ac15eb70 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b358f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ac0d3790 con 0x7fc6ac15e790 2024-04-15T15:22:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b358f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ac0d45b0 con 0x7fc6ac0d1cf0 2024-04-15T15:22:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b132b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6ac15e790 0x7fc6ac0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b132b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6ac15e790 0x7fc6ac0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52946/0 (socket says 172.21.15.12:52946) 2024-04-15T15:22:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b132b640 1 -- 172.21.15.12:0/695760272 learned_addr learned my addr 172.21.15.12:0/695760272 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b132b640 1 -- 172.21.15.12:0/695760272 >> v2:172.21.15.55:3300/0 conn(0x7fc6ac0d1cf0 msgr2=0x7fc6ac0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b132b640 1 --2- 172.21.15.12:0/695760272 >> v2:172.21.15.55:3300/0 conn(0x7fc6ac0d1cf0 0x7fc6ac0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b132b640 1 -- 172.21.15.12:0/695760272 >> v1:172.21.15.55:6789/0 conn(0x7fc6ac0b7be0 legacy=0x7fc6ac0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.251+0000 7fc6b132b640 1 -- 172.21.15.12:0/695760272 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac0d3c80 con 0x7fc6ac15e790 2024-04-15T15:22:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b132b640 1 --2- 172.21.15.12:0/695760272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac15e790 0x7fc6ac0b76a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0d56e0 tx=0x7fc69c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23159b6c32f57df server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6abfff640 1 -- 172.21.15.12:0/695760272 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c02c950 con 0x7fc6ac15e790 2024-04-15T15:22:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6abfff640 1 -- 172.21.15.12:0/695760272 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc69c02caf0 con 0x7fc6ac15e790 2024-04-15T15:22:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6abfff640 1 -- 172.21.15.12:0/695760272 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0343f0 con 0x7fc6ac15e790 2024-04-15T15:22:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b358f640 1 -- 172.21.15.12:0/695760272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac15e790 msgr2=0x7fc6ac0b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b358f640 1 --2- 172.21.15.12:0/695760272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac15e790 0x7fc6ac0b76a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0d56e0 tx=0x7fc69c02bdb0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b358f640 1 -- 172.21.15.12:0/695760272 shutdown_connections 2024-04-15T15:22:07.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b358f640 1 --2- 172.21.15.12:0/695760272 >> v2:172.21.15.55:3300/0 conn(0x7fc6ac0d1cf0 0x7fc6ac0d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b358f640 1 --2- 172.21.15.12:0/695760272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac15e790 0x7fc6ac0b76a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b358f640 1 -- 172.21.15.12:0/695760272 >> 172.21.15.12:0/695760272 conn(0x7fc6ac067b40 msgr2=0x7fc6ac067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b358f640 1 -- 172.21.15.12:0/695760272 shutdown_connections 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.252+0000 7fc6b358f640 1 -- 172.21.15.12:0/695760272 wait complete. 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b358f640 1 Processor -- start 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b358f640 1 -- start start 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b358f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac0b7be0 0x7fc6ac1f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b358f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6ac0d1cf0 0x7fc6ac0c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b358f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6ac15e790 0x7fc6ac157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b358f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac0d4f80 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b358f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac0d4240 con 0x7fc6ac15e790 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b358f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ac0d5100 con 0x7fc6ac0d1cf0 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b132b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac0b7be0 0x7fc6ac1f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b132b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac0b7be0 0x7fc6ac1f52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52966/0 (socket says 172.21.15.12:52966) 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.254+0000 7fc6b132b640 1 -- 172.21.15.12:0/2016510196 learned_addr learned my addr 172.21.15.12:0/2016510196 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b132b640 1 -- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6ac0d1cf0 msgr2=0x7fc6ac0c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b132b640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6ac0d1cf0 0x7fc6ac0c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b132b640 1 -- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6ac15e790 msgr2=0x7fc6ac157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b1b2c640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6ac15e790 0x7fc6ac157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b132b640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6ac15e790 0x7fc6ac157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b132b640 1 -- 172.21.15.12:0/2016510196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac0c93c0 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b1b2c640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6ac15e790 0x7fc6ac157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b132b640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac0b7be0 0x7fc6ac1f52a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0093e0 tx=0x7fc69c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6a9ffb640 1 -- 172.21.15.12:0/2016510196 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c043020 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6a9ffb640 1 -- 172.21.15.12:0/2016510196 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc69c03e450 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ac0c9680 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6a9ffb640 1 -- 172.21.15.12:0/2016510196 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c03e840 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.255+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ac0c9c40 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.256+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ac0689e0 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.257+0000 7fc6a9ffb640 1 -- 172.21.15.12:0/2016510196 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc69c046020 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.257+0000 7fc6a9ffb640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc69806c8f0 0x7fc69806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.257+0000 7fc6a9ffb640 1 -- 172.21.15.12:0/2016510196 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fc69c0ae950 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.257+0000 7fc6b0b2a640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc69806c8f0 0x7fc69806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.258+0000 7fc6b0b2a640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc69806c8f0 0x7fc69806edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0cd260 tx=0x7fc6a0007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.262+0000 7fc6a9ffb640 1 -- 172.21.15.12:0/2016510196 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc69c04d030 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.460+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc6ac0cb420 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.461+0000 7fc6a9ffb640 1 -- 172.21.15.12:0/2016510196 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fc69c07c9d0 con 0x7fc6ac0b7be0 2024-04-15T15:22:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc69806c8f0 msgr2=0x7fc69806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc69806c8f0 0x7fc69806edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0cd260 tx=0x7fc6a0007990 comp rx=0 tx=0).stop 2024-04-15T15:22:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac0b7be0 msgr2=0x7fc6ac1f52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac0b7be0 0x7fc6ac1f52a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0093e0 tx=0x7fc69c004a40 comp rx=0 tx=0).stop 2024-04-15T15:22:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 shutdown_connections 2024-04-15T15:22:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc69806c8f0 0x7fc69806edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6ac15e790 0x7fc6ac157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6ac0d1cf0 0x7fc6ac0c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 --2- 172.21.15.12:0/2016510196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6ac0b7be0 0x7fc6ac1f52a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 >> 172.21.15.12:0/2016510196 conn(0x7fc6ac067b40 msgr2=0x7fc6ac0b94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 shutdown_connections 2024-04-15T15:22:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.463+0000 7fc6b358f640 1 -- 172.21.15.12:0/2016510196 wait complete. 2024-04-15T15:22:07.475 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:22:07.475 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:07.495 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.598+0000 7f267d02a640 1 Processor -- start 2024-04-15T15:22:07.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.598+0000 7f267d02a640 1 -- start start 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f267d02a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26780701f0 0x7f2678160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f267d02a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f267815fb30 0x7f26780d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f267d02a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2678070720 con 0x7f267809a9d0 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f267d02a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f267809ba90 con 0x7f26780701f0 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f267d02a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f26780d08c0 con 0x7f267815fb30 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f266ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26780701f0 0x7f2678160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f266ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26780701f0 0x7f2678160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52968/0 (socket says 172.21.15.12:52968) 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f266ffff640 1 -- 172.21.15.12:0/2593816728 learned_addr learned my addr 172.21.15.12:0/2593816728 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f266ffff640 1 -- 172.21.15.12:0/2593816728 >> v2:172.21.15.55:3300/0 conn(0x7f267815fb30 msgr2=0x7f26780d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f266ffff640 1 --2- 172.21.15.12:0/2593816728 >> v2:172.21.15.55:3300/0 conn(0x7f267815fb30 0x7f26780d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f266ffff640 1 -- 172.21.15.12:0/2593816728 >> v1:172.21.15.55:6789/0 conn(0x7f267809a9d0 legacy=0x7f267809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:07.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.599+0000 7f266ffff640 1 -- 172.21.15.12:0/2593816728 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2678070a90 con 0x7f26780701f0 2024-04-15T15:22:07.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.600+0000 7f266ffff640 1 --2- 172.21.15.12:0/2593816728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26780701f0 0x7f2678160420 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2678073740 tx=0x7f266002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3865c490e101c210 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.600+0000 7f267659c640 1 -- 172.21.15.12:0/2593816728 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660035020 con 0x7f26780701f0 2024-04-15T15:22:07.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.600+0000 7f267659c640 1 -- 172.21.15.12:0/2593816728 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2660031870 con 0x7f26780701f0 2024-04-15T15:22:07.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.600+0000 7f267659c640 1 -- 172.21.15.12:0/2593816728 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660031c40 con 0x7f26780701f0 2024-04-15T15:22:07.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.601+0000 7f267d02a640 1 -- 172.21.15.12:0/2593816728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26780701f0 msgr2=0x7f2678160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.601+0000 7f267d02a640 1 --2- 172.21.15.12:0/2593816728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26780701f0 0x7f2678160420 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2678073740 tx=0x7f266002b910 comp rx=0 tx=0).stop 2024-04-15T15:22:07.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.601+0000 7f267d02a640 1 -- 172.21.15.12:0/2593816728 shutdown_connections 2024-04-15T15:22:07.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.601+0000 7f267d02a640 1 --2- 172.21.15.12:0/2593816728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26780701f0 0x7f2678160420 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.601+0000 7f267d02a640 1 --2- 172.21.15.12:0/2593816728 >> v2:172.21.15.55:3300/0 conn(0x7f267815fb30 0x7f26780d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.601+0000 7f267d02a640 1 -- 172.21.15.12:0/2593816728 >> 172.21.15.12:0/2593816728 conn(0x7f26780d2460 msgr2=0x7f26780d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:07.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.601+0000 7f267d02a640 1 -- 172.21.15.12:0/2593816728 shutdown_connections 2024-04-15T15:22:07.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.601+0000 7f267d02a640 1 -- 172.21.15.12:0/2593816728 wait complete. 2024-04-15T15:22:07.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.602+0000 7f267d02a640 1 Processor -- start 2024-04-15T15:22:07.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.602+0000 7f267d02a640 1 -- start start 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267d02a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26780701f0 0x7f26780c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267d02a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f267809a9d0 0x7f267809fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267d02a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f267815fb30 0x7f26780c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267d02a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f267815ffc0 con 0x7f267815fb30 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267d02a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2678071050 con 0x7f26780701f0 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267d02a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f26780b1c60 con 0x7f267809a9d0 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f267815fb30 0x7f26780c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f267815fb30 0x7f26780c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52976/0 (socket says 172.21.15.12:52976) 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 -- 172.21.15.12:0/2582047540 learned_addr learned my addr 172.21.15.12:0/2582047540 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f2676d9d640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26780701f0 0x7f26780c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f266ffff640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f267809a9d0 0x7f267809fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 -- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f267809a9d0 msgr2=0x7f267809fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f267809a9d0 0x7f267809fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 -- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26780701f0 msgr2=0x7f26780c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26780701f0 0x7f26780c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 -- 172.21.15.12:0/2582047540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26780c92a0 con 0x7f267815fb30 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f2676d9d640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26780701f0 0x7f26780c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267759e640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f267815fb30 0x7f26780c7ab0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f267000b970 tx=0x7f267000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f266f7fe640 1 -- 172.21.15.12:0/2582047540 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670019070 con 0x7f267815fb30 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f266f7fe640 1 -- 172.21.15.12:0/2582047540 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267000cb00 con 0x7f267815fb30 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f266f7fe640 1 -- 172.21.15.12:0/2582047540 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670014440 con 0x7f267815fb30 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f267809ed40 con 0x7f267815fb30 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.603+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f267809f300 con 0x7f267815fb30 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.604+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26780689e0 con 0x7f267815fb30 2024-04-15T15:22:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.605+0000 7f266f7fe640 1 -- 172.21.15.12:0/2582047540 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f267001c070 con 0x7f267815fb30 2024-04-15T15:22:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.606+0000 7f266f7fe640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f265806c8f0 0x7f265806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.606+0000 7f266f7fe640 1 -- 172.21.15.12:0/2582047540 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f267008bcc0 con 0x7f267815fb30 2024-04-15T15:22:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.606+0000 7f2676d9d640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f265806c8f0 0x7f265806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.606+0000 7f2676d9d640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f265806c8f0 0x7f265806edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2668001750 tx=0x7f2668009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.610+0000 7f266f7fe640 1 -- 172.21.15.12:0/2582047540 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2670059cb0 con 0x7f267815fb30 2024-04-15T15:22:07.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.798+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26780d3370 con 0x7f267815fb30 2024-04-15T15:22:07.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.799+0000 7f266f7fe640 1 -- 172.21.15.12:0/2582047540 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f26700595b0 con 0x7f267815fb30 2024-04-15T15:22:07.801 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:07.801 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:07.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f265806c8f0 msgr2=0x7f265806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f265806c8f0 0x7f265806edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2668001750 tx=0x7f2668009040 comp rx=0 tx=0).stop 2024-04-15T15:22:07.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f267815fb30 msgr2=0x7f26780c7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f267815fb30 0x7f26780c7ab0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f267000b970 tx=0x7f267000be30 comp rx=0 tx=0).stop 2024-04-15T15:22:07.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 shutdown_connections 2024-04-15T15:22:07.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f267815fb30 0x7f26780c7ab0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f267809a9d0 0x7f267809fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f265806c8f0 0x7f265806edb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 --2- 172.21.15.12:0/2582047540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26780701f0 0x7f26780c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 >> 172.21.15.12:0/2582047540 conn(0x7f26780d2460 msgr2=0x7f26780b6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:07.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 shutdown_connections 2024-04-15T15:22:07.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.802+0000 7f267d02a640 1 -- 172.21.15.12:0/2582047540 wait complete. 2024-04-15T15:22:07.814 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:07.815 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:07.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.921+0000 7f1ab350e640 1 Processor -- start 2024-04-15T15:22:07.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.922+0000 7f1ab350e640 1 -- start start 2024-04-15T15:22:07.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.922+0000 7f1ab350e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 0x7f1aac0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.922+0000 7f1ab350e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1aac0d1cf0 0x7f1aac0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.922+0000 7f1ab350e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aac15eb70 con 0x7f1aac0b7be0 2024-04-15T15:22:07.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.922+0000 7f1ab350e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1aac0d3790 con 0x7f1aac0d1cf0 2024-04-15T15:22:07.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.922+0000 7f1ab350e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1aac0d45b0 con 0x7f1aac15e790 2024-04-15T15:22:07.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab0aa9640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1aac0b7be0 0x7f1aac0d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34736/0 (socket says 172.21.15.12:34736) 2024-04-15T15:22:07.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab0aa9640 1 -- 172.21.15.12:0/363997253 learned_addr learned my addr 172.21.15.12:0/363997253 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:07.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab1aab640 1 --2- 172.21.15.12:0/363997253 >> v2:172.21.15.55:3300/0 conn(0x7f1aac0d1cf0 0x7f1aac0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab12aa640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 0x7f1aac0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 631151869 0 0) 0x7f1aac15eb70 con 0x7f1aac0b7be0 2024-04-15T15:22:07.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a98003520 con 0x7f1aac0b7be0 2024-04-15T15:22:07.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3393220709 0 0) 0x7f1a98003520 con 0x7f1aac0b7be0 2024-04-15T15:22:07.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 msgr2=0x7f1aac0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:07.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 0x7f1aac0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 >> v2:172.21.15.55:3300/0 conn(0x7f1aac0d1cf0 msgr2=0x7f1aac0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:07.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 --2- 172.21.15.12:0/363997253 >> v2:172.21.15.55:3300/0 conn(0x7f1aac0d1cf0 0x7f1aac0d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aac0d3c80 con 0x7f1aac0b7be0 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f1a9c001690 con 0x7f1aac0b7be0 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab12aa640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 0x7f1aac0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab1aab640 1 --2- 172.21.15.12:0/363997253 >> v2:172.21.15.55:3300/0 conn(0x7f1aac0d1cf0 0x7f1aac0d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 >> v1:172.21.15.12:6789/0 conn(0x7f1aac0b7be0 legacy=0x7f1aac0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a980039b0 0x7f1a98003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a98004800 0x7f1a98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 0x7f1a980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac0d3790 con 0x7f1a98004800 2024-04-15T15:22:07.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac0d45b0 con 0x7f1a980039b0 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9800ba40 con 0x7f1aac15e790 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab0aa9640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a98004800 0x7f1a98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab12aa640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a980039b0 0x7f1a98003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.923+0000 7f1ab1aab640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 0x7f1a980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.924+0000 7f1ab0aa9640 1 -- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 msgr2=0x7f1a980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.924+0000 7f1ab0aa9640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac15e790 0x7f1a980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.924+0000 7f1ab0aa9640 1 -- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a980039b0 msgr2=0x7f1a98003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.924+0000 7f1ab0aa9640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a980039b0 0x7f1a98003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.924+0000 7f1ab0aa9640 1 -- 172.21.15.12:0/363997253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a98003520 con 0x7f1a98004800 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.924+0000 7f1ab0aa9640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a98004800 0x7f1a98004c80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c007fd0 tx=0x7f1a98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c818f3fe9dce76 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.924+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c007d50 con 0x7f1a98004800 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a9c030070 con 0x7f1a98004800 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1aa3fff640 1 -- 172.21.15.12:0/363997253 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c005d10 con 0x7f1a98004800 2024-04-15T15:22:07.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1ab350e640 1 -- 172.21.15.12:0/363997253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a98004800 msgr2=0x7f1a98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1ab350e640 1 --2- 172.21.15.12:0/363997253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a98004800 0x7f1a98004c80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c007fd0 tx=0x7f1a98005830 comp rx=0 tx=0).stop 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1ab1aab640 1 -- 172.21.15.12:0/363997253 reap_dead start 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1ab350e640 1 -- 172.21.15.12:0/363997253 shutdown_connections 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1ab350e640 1 -- 172.21.15.12:0/363997253 >> 172.21.15.12:0/363997253 conn(0x7f1aac067b40 msgr2=0x7f1aac067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1ab350e640 1 -- 172.21.15.12:0/363997253 shutdown_connections 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.925+0000 7f1ab350e640 1 -- 172.21.15.12:0/363997253 wait complete. 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.926+0000 7f1ab350e640 1 Processor -- start 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.926+0000 7f1ab350e640 1 -- start start 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab350e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a98004800 0x7f1aac154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab350e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1aac160bb0 0x7f1aac161030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab350e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac0aca40 0x7f1aac0acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab350e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac0b76a0 con 0x7f1aac160bb0 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab350e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac0d4240 con 0x7f1a98004800 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab350e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1aac0b7820 con 0x7f1aac0aca40 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab0aa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1aac160bb0 0x7f1aac161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab0aa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1aac160bb0 0x7f1aac161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52994/0 (socket says 172.21.15.12:52994) 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab0aa9640 1 -- 172.21.15.12:0/113167052 learned_addr learned my addr 172.21.15.12:0/113167052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab12aa640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a98004800 0x7f1aac154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab1aab640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac0aca40 0x7f1aac0acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab0aa9640 1 -- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac0aca40 msgr2=0x7f1aac0acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab0aa9640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac0aca40 0x7f1aac0acec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab0aa9640 1 -- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a98004800 msgr2=0x7f1aac154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab0aa9640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a98004800 0x7f1aac154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:07.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.927+0000 7f1ab0aa9640 1 -- 172.21.15.12:0/113167052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aac0ad620 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.928+0000 7f1ab1aab640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac0aca40 0x7f1aac0acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.928+0000 7f1ab0aa9640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1aac160bb0 0x7f1aac161030 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c02c540 tx=0x7f1aac0d3e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.928+0000 7f1aa1ffb640 1 -- 172.21.15.12:0/113167052 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c045070 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.928+0000 7f1aa1ffb640 1 -- 172.21.15.12:0/113167052 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a9c004c10 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.928+0000 7f1aa1ffb640 1 -- 172.21.15.12:0/113167052 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c007d50 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.928+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aac0ad900 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.928+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aac0adec0 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.929+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aac0689e0 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.930+0000 7f1aa1ffb640 1 -- 172.21.15.12:0/113167052 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a9c007070 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.930+0000 7f1aa1ffb640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1a9006c920 0x7f1a9006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.930+0000 7f1aa1ffb640 1 -- 172.21.15.12:0/113167052 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f1a9c0acdf0 con 0x7f1aac160bb0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.930+0000 7f1ab12aa640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1a9006c920 0x7f1a9006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:07.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.931+0000 7f1ab12aa640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1a9006c920 0x7f1a9006ede0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009a50 tx=0x7f1aa8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:07.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:07.934+0000 7f1aa1ffb640 1 -- 172.21.15.12:0/113167052 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a9c07a870 con 0x7f1aac160bb0 2024-04-15T15:22:08.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.120+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1aac15f640 con 0x7f1aac160bb0 2024-04-15T15:22:08.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.120+0000 7f1aa1ffb640 1 -- 172.21.15.12:0/113167052 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f1a9c07eb70 con 0x7f1aac160bb0 2024-04-15T15:22:08.122 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1a9006c920 msgr2=0x7f1a9006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1a9006c920 0x7f1a9006ede0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009a50 tx=0x7f1aa8009190 comp rx=0 tx=0).stop 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1aac160bb0 msgr2=0x7f1aac161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1aac160bb0 0x7f1aac161030 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c02c540 tx=0x7f1aac0d3e50 comp rx=0 tx=0).stop 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 shutdown_connections 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1aac0aca40 0x7f1aac0acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1aac160bb0 0x7f1aac161030 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1a9006c920 0x7f1a9006ede0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 --2- 172.21.15.12:0/113167052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a98004800 0x7f1aac154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 >> 172.21.15.12:0/113167052 conn(0x7f1aac067b40 msgr2=0x7f1aac0cdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 shutdown_connections 2024-04-15T15:22:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.122+0000 7f1ab350e640 1 -- 172.21.15.12:0/113167052 wait complete. 2024-04-15T15:22:08.134 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:08.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.241+0000 7f1738441640 1 Processor -- start 2024-04-15T15:22:08.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.241+0000 7f1738441640 1 -- start start 2024-04-15T15:22:08.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f1738441640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f17300d3240 0x7f17300d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f1738441640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f17300d1440 0x7f17300b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f1738441640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17300d2480 con 0x7f173015e790 2024-04-15T15:22:08.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f1738441640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f17300d2db0 con 0x7f17300d1440 2024-04-15T15:22:08.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f1738441640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f17300d2760 con 0x7f17300d3240 2024-04-15T15:22:08.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f17359dc640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f17300d3240 0x7f17300d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f17369de640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f17300d1440 0x7f17300b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f17369de640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f17300d1440 0x7f17300b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:42904/0 (socket says 172.21.15.12:42904) 2024-04-15T15:22:08.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f17369de640 1 -- 172.21.15.12:0/2477224740 learned_addr learned my addr 172.21.15.12:0/2477224740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:08.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.242+0000 7f17361dd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f173015e790 0x7f17300cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:60332/0 (socket says 172.21.15.12:60332) 2024-04-15T15:22:08.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.243+0000 7f17369de640 1 -- 172.21.15.12:0/2477224740 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f17300d3240 msgr2=0x7f17300d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:08.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.243+0000 7f17369de640 1 --2- 172.21.15.12:0/2477224740 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f17300d3240 0x7f17300d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.243+0000 7f17369de640 1 -- 172.21.15.12:0/2477224740 >> v1:172.21.15.55:6789/0 conn(0x7f173015e790 legacy=0x7f17300cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.243+0000 7f17369de640 1 -- 172.21.15.12:0/2477224740 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17300d2a40 con 0x7f17300d1440 2024-04-15T15:22:08.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.243+0000 7f17359dc640 1 --2- 172.21.15.12:0/2477224740 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f17300d3240 0x7f17300d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:08.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.244+0000 7f17369de640 1 --2- 172.21.15.12:0/2477224740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 0x7f17300b74f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1720009e50 tx=0x7f172002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dea084bc7e5648d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.244+0000 7f17351db640 1 -- 172.21.15.12:0/2477224740 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172002cb00 con 0x7f17300d1440 2024-04-15T15:22:08.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.244+0000 7f17351db640 1 -- 172.21.15.12:0/2477224740 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f172002cca0 con 0x7f17300d1440 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.244+0000 7f17351db640 1 -- 172.21.15.12:0/2477224740 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720034610 con 0x7f17300d1440 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.244+0000 7f1738441640 1 -- 172.21.15.12:0/2477224740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 msgr2=0x7f17300b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.244+0000 7f1738441640 1 --2- 172.21.15.12:0/2477224740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 0x7f17300b74f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1720009e50 tx=0x7f172002bf60 comp rx=0 tx=0).stop 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.245+0000 7f1738441640 1 -- 172.21.15.12:0/2477224740 shutdown_connections 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.245+0000 7f1738441640 1 --2- 172.21.15.12:0/2477224740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 0x7f17300b74f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.245+0000 7f1738441640 1 --2- 172.21.15.12:0/2477224740 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f17300d3240 0x7f17300d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.245+0000 7f1738441640 1 -- 172.21.15.12:0/2477224740 >> 172.21.15.12:0/2477224740 conn(0x7f173008bf60 msgr2=0x7f17300d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.245+0000 7f1738441640 1 -- 172.21.15.12:0/2477224740 shutdown_connections 2024-04-15T15:22:08.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.245+0000 7f1738441640 1 -- 172.21.15.12:0/2477224740 wait complete. 2024-04-15T15:22:08.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.246+0000 7f1738441640 1 Processor -- start 2024-04-15T15:22:08.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.246+0000 7f1738441640 1 -- start start 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.246+0000 7f1738441640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 0x7f173009fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.246+0000 7f1738441640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d3240 0x7f17301f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.246+0000 7f1738441640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f173015e790 0x7f17300a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.246+0000 7f1738441640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f17300d5180 con 0x7f17300d3240 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.246+0000 7f1738441640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f17300d0430 con 0x7f17300d1440 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.246+0000 7f1738441640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f17300d5300 con 0x7f173015e790 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d3240 0x7f17301f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d3240 0x7f17301f8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53008/0 (socket says 172.21.15.12:53008) 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 -- 172.21.15.12:0/1546484259 learned_addr learned my addr 172.21.15.12:0/1546484259 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17361dd640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 0x7f173009fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17369de640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f173015e790 0x7f17300a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 -- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f173015e790 msgr2=0x7f17300a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f173015e790 0x7f17300a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 -- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 msgr2=0x7f173009fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 0x7f173009fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 -- 172.21.15.12:0/1546484259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17301f8e90 con 0x7f17300d3240 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f17359dc640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d3240 0x7f17301f8950 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f172400ec10 tx=0x7f172400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f172effd640 1 -- 172.21.15.12:0/1546484259 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724017070 con 0x7f17300d3240 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f172effd640 1 -- 172.21.15.12:0/1546484259 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f172400cc30 con 0x7f17300d3240 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173009ec40 con 0x7f17300d3240 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f172effd640 1 -- 172.21.15.12:0/1546484259 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724013470 con 0x7f17300d3240 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.247+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f173009f200 con 0x7f17300d3240 2024-04-15T15:22:08.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.248+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17300689e0 con 0x7f17300d3240 2024-04-15T15:22:08.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.249+0000 7f172effd640 1 -- 172.21.15.12:0/1546484259 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f172401a070 con 0x7f17300d3240 2024-04-15T15:22:08.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.249+0000 7f172effd640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f171c06c8f0 0x7f171c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.249+0000 7f172effd640 1 -- 172.21.15.12:0/1546484259 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f172408c1d0 con 0x7f17300d3240 2024-04-15T15:22:08.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.250+0000 7f17361dd640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f171c06c8f0 0x7f171c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.250+0000 7f17361dd640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f171c06c8f0 0x7f171c06edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f173015d950 tx=0x7f1728007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.254+0000 7f172effd640 1 -- 172.21.15.12:0/1546484259 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f17301f8e90 con 0x7f17300d3240 2024-04-15T15:22:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.436+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f17300cb2b0 con 0x7f17300d3240 2024-04-15T15:22:08.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.436+0000 7f172effd640 1 -- 172.21.15.12:0/1546484259 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f1724059b80 con 0x7f17300d3240 2024-04-15T15:22:08.438 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f171c06c8f0 msgr2=0x7f171c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f171c06c8f0 0x7f171c06edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f173015d950 tx=0x7f1728007e80 comp rx=0 tx=0).stop 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d3240 msgr2=0x7f17301f8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d3240 0x7f17301f8950 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f172400ec10 tx=0x7f172400c520 comp rx=0 tx=0).stop 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 shutdown_connections 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f173015e790 0x7f17300a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17300d3240 0x7f17301f8950 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f171c06c8f0 0x7f171c06edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 --2- 172.21.15.12:0/1546484259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17300d1440 0x7f173009fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 >> 172.21.15.12:0/1546484259 conn(0x7f173008bf60 msgr2=0x7f17300b8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:08.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 shutdown_connections 2024-04-15T15:22:08.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.438+0000 7f1738441640 1 -- 172.21.15.12:0/1546484259 wait complete. 2024-04-15T15:22:08.451 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.557+0000 7f59c6d9d640 1 Processor -- start 2024-04-15T15:22:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.557+0000 7f59c6d9d640 1 -- start start 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c6d9d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c6d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f59c00d2cc0 0x7f59c00d3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c6d9d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59c00d02e0 con 0x7f59c00d0d40 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c6d9d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f59c00d1a00 con 0x7f59c00d2cc0 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c6d9d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f59c00d1f90 con 0x7f59c00d4e60 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c659c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f59c00d2cc0 0x7f59c00d3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c559a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c5d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f59c00d0d40 0x7f59c00d1120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:60336/0 (socket says 172.21.15.12:60336) 2024-04-15T15:22:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c659c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f59c00d2cc0 0x7f59c00d3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:42924/0 (socket says 172.21.15.12:42924) 2024-04-15T15:22:08.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.558+0000 7f59c5d9b640 1 -- 172.21.15.12:0/3632184653 learned_addr learned my addr 172.21.15.12:0/3632184653 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:08.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.559+0000 7f59c559a640 1 -- 172.21.15.12:0/3632184653 >> v2:172.21.15.55:3300/0 conn(0x7f59c00d2cc0 msgr2=0x7f59c00d3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:08.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.559+0000 7f59c559a640 1 --2- 172.21.15.12:0/3632184653 >> v2:172.21.15.55:3300/0 conn(0x7f59c00d2cc0 0x7f59c00d3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.559+0000 7f59c559a640 1 -- 172.21.15.12:0/3632184653 >> v1:172.21.15.55:6789/0 conn(0x7f59c00d0d40 legacy=0x7f59c00d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.559+0000 7f59c4d99640 1 -- 172.21.15.12:0/3632184653 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2800663799 0 0) 0x7f59c00d02e0 con 0x7f59c00d0d40 2024-04-15T15:22:08.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.559+0000 7f59c559a640 1 -- 172.21.15.12:0/3632184653 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c00d21e0 con 0x7f59c00d4e60 2024-04-15T15:22:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c559a640 1 --2- 172.21.15.12:0/3632184653 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00cfb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009790 tx=0x7f59b402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbb5292b7672fd66 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c4d99640 1 -- 172.21.15.12:0/3632184653 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4039070 con 0x7f59c00d4e60 2024-04-15T15:22:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c4d99640 1 -- 172.21.15.12:0/3632184653 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b40316f0 con 0x7f59c00d4e60 2024-04-15T15:22:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c4d99640 1 -- 172.21.15.12:0/3632184653 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4031ac0 con 0x7f59c00d4e60 2024-04-15T15:22:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3632184653 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 msgr2=0x7f59c00cfb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3632184653 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00cfb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009790 tx=0x7f59b402b790 comp rx=0 tx=0).stop 2024-04-15T15:22:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3632184653 shutdown_connections 2024-04-15T15:22:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3632184653 >> v2:172.21.15.55:3300/0 conn(0x7f59c00d2cc0 0x7f59c00d3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3632184653 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00cfb90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3632184653 >> 172.21.15.12:0/3632184653 conn(0x7f59c008bf60 msgr2=0x7f59c00d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3632184653 shutdown_connections 2024-04-15T15:22:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.560+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3632184653 wait complete. 2024-04-15T15:22:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.561+0000 7f59c6d9d640 1 Processor -- start 2024-04-15T15:22:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c6d9d640 1 -- start start 2024-04-15T15:22:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c6d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59c00d0d40 0x7f59c014a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c6d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c00d2cc0 0x7f59c01614a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c6d9d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00b8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c6d9d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f59c015fcc0 con 0x7f59c00d2cc0 2024-04-15T15:22:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c6d9d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f59c00d27a0 con 0x7f59c00d0d40 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c6d9d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f59c015fe40 con 0x7f59c00d4e60 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c559a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c00d2cc0 0x7f59c01614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c559a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c00d2cc0 0x7f59c01614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53022/0 (socket says 172.21.15.12:53022) 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.562+0000 7f59c559a640 1 -- 172.21.15.12:0/3198137598 learned_addr learned my addr 172.21.15.12:0/3198137598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c5d9b640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59c00d0d40 0x7f59c014a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c659c640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c559a640 1 -- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 msgr2=0x7f59c00b8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c559a640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00b8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c559a640 1 -- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59c00d0d40 msgr2=0x7f59c014a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c559a640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59c00d0d40 0x7f59c014a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c559a640 1 -- 172.21.15.12:0/3198137598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c014ae00 con 0x7f59c00d2cc0 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c559a640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c00d2cc0 0x7f59c01614a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f59b4037040 tx=0x7f59b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59be7fc640 1 -- 172.21.15.12:0/3198137598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4039040 con 0x7f59c00d2cc0 2024-04-15T15:22:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59be7fc640 1 -- 172.21.15.12:0/3198137598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b4004020 con 0x7f59c00d2cc0 2024-04-15T15:22:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c01e80d0 con 0x7f59c00d2cc0 2024-04-15T15:22:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59be7fc640 1 -- 172.21.15.12:0/3198137598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4004430 con 0x7f59c00d2cc0 2024-04-15T15:22:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.563+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c01e8690 con 0x7f59c00d2cc0 2024-04-15T15:22:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.564+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c004fd50 con 0x7f59c00d2cc0 2024-04-15T15:22:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.565+0000 7f59be7fc640 1 -- 172.21.15.12:0/3198137598 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f59b403c040 con 0x7f59c00d2cc0 2024-04-15T15:22:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.565+0000 7f59be7fc640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f59ac06c8f0 0x7f59ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.565+0000 7f59be7fc640 1 -- 172.21.15.12:0/3198137598 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f59b40ad9d0 con 0x7f59c00d2cc0 2024-04-15T15:22:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.565+0000 7f59c5d9b640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f59ac06c8f0 0x7f59ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.566+0000 7f59c5d9b640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f59ac06c8f0 0x7f59ac06edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f59b00012e0 tx=0x7f59b0001630 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.570+0000 7f59be7fc640 1 -- 172.21.15.12:0/3198137598 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f59b404d030 con 0x7f59c00d2cc0 2024-04-15T15:22:08.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.755+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f59c004fd50 con 0x7f59c00d2cc0 2024-04-15T15:22:08.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.756+0000 7f59be7fc640 1 -- 172.21.15.12:0/3198137598 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f59b4044030 con 0x7f59c00d2cc0 2024-04-15T15:22:08.757 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.757+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f59ac06c8f0 msgr2=0x7f59ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.757+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f59ac06c8f0 0x7f59ac06edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f59b00012e0 tx=0x7f59b0001630 comp rx=0 tx=0).stop 2024-04-15T15:22:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.757+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c00d2cc0 msgr2=0x7f59c01614a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.757+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c00d2cc0 0x7f59c01614a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f59b4037040 tx=0x7f59b40046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.758+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 shutdown_connections 2024-04-15T15:22:08.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.758+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f59c00d4e60 0x7f59c00b8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.758+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c00d2cc0 0x7f59c01614a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.758+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f59ac06c8f0 0x7f59ac06edb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.758+0000 7f59c6d9d640 1 --2- 172.21.15.12:0/3198137598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59c00d0d40 0x7f59c014a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.758+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 >> 172.21.15.12:0/3198137598 conn(0x7f59c008bf60 msgr2=0x7f59c006ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.758+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 shutdown_connections 2024-04-15T15:22:08.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.758+0000 7f59c6d9d640 1 -- 172.21.15.12:0/3198137598 wait complete. 2024-04-15T15:22:08.770 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.876+0000 7f65c68d3640 1 Processor -- start 2024-04-15T15:22:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.876+0000 7f65c68d3640 1 -- start start 2024-04-15T15:22:08.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65c68d3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f65c00d15c0 0x7f65c006cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65c68d3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f65c006d050 0x7f65c006d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65c68d3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c00d0530 con 0x7f65c015e7a0 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65c68d3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f65c00cf490 con 0x7f65c00d15c0 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65c68d3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f65c00cf770 con 0x7f65c006d050 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65b7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f65c00d15c0 0x7f65c006cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65b7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f65c00d15c0 0x7f65c006cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53028/0 (socket says 172.21.15.12:53028) 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65b7fff640 1 -- 172.21.15.12:0/4126260556 learned_addr learned my addr 172.21.15.12:0/4126260556 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65c4e70640 1 --2- 172.21.15.12:0/4126260556 >> v2:172.21.15.55:3300/0 conn(0x7f65c006d050 0x7f65c006d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65b7fff640 1 -- 172.21.15.12:0/4126260556 >> v2:172.21.15.55:3300/0 conn(0x7f65c006d050 msgr2=0x7f65c006d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65b7fff640 1 --2- 172.21.15.12:0/4126260556 >> v2:172.21.15.55:3300/0 conn(0x7f65c006d050 0x7f65c006d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65b7fff640 1 -- 172.21.15.12:0/4126260556 >> v1:172.21.15.55:6789/0 conn(0x7f65c015e7a0 legacy=0x7f65c00d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65b7fff640 1 -- 172.21.15.12:0/4126260556 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c00cfa50 con 0x7f65c00d15c0 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.877+0000 7f65c4e70640 1 --2- 172.21.15.12:0/4126260556 >> v2:172.21.15.55:3300/0 conn(0x7f65c006d050 0x7f65c006d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:08.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.878+0000 7f65b7fff640 1 --2- 172.21.15.12:0/4126260556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c006cb10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f65c00d2010 tx=0x7f65a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26ad6c97ecf601 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.878+0000 7f65bf7fe640 1 -- 172.21.15.12:0/4126260556 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a802e070 con 0x7f65c00d15c0 2024-04-15T15:22:08.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.878+0000 7f65bf7fe640 1 -- 172.21.15.12:0/4126260556 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a8037070 con 0x7f65c00d15c0 2024-04-15T15:22:08.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.878+0000 7f65bf7fe640 1 -- 172.21.15.12:0/4126260556 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8031a00 con 0x7f65c00d15c0 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.879+0000 7f65c68d3640 1 -- 172.21.15.12:0/4126260556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 msgr2=0x7f65c006cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.879+0000 7f65c68d3640 1 --2- 172.21.15.12:0/4126260556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c006cb10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f65c00d2010 tx=0x7f65a802b880 comp rx=0 tx=0).stop 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.879+0000 7f65c68d3640 1 -- 172.21.15.12:0/4126260556 shutdown_connections 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.879+0000 7f65c68d3640 1 --2- 172.21.15.12:0/4126260556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c006cb10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.879+0000 7f65c68d3640 1 --2- 172.21.15.12:0/4126260556 >> v2:172.21.15.55:3300/0 conn(0x7f65c006d050 0x7f65c006d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.879+0000 7f65c68d3640 1 -- 172.21.15.12:0/4126260556 >> 172.21.15.12:0/4126260556 conn(0x7f65c008bf60 msgr2=0x7f65c00d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.879+0000 7f65c68d3640 1 -- 172.21.15.12:0/4126260556 shutdown_connections 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.879+0000 7f65c68d3640 1 -- 172.21.15.12:0/4126260556 wait complete. 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.880+0000 7f65c68d3640 1 Processor -- start 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.880+0000 7f65c68d3640 1 -- start start 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65c68d3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f65c006d050 0x7f65c014e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65c68d3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c00c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65c68d3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c015e7a0 0x7f65c00b6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65c68d3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f65c015eb80 con 0x7f65c00d15c0 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65c68d3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f65c00cf180 con 0x7f65c015e7a0 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65c68d3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f65c00d1a40 con 0x7f65c006d050 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c00c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c00c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53044/0 (socket says 172.21.15.12:53044) 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 -- 172.21.15.12:0/3361624524 learned_addr learned my addr 172.21.15.12:0/3361624524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65c4e70640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c015e7a0 0x7f65c00b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65bffff640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f65c006d050 0x7f65c014e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 -- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f65c006d050 msgr2=0x7f65c014e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f65c006d050 0x7f65c014e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 -- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c015e7a0 msgr2=0x7f65c00b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c015e7a0 0x7f65c00b6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 -- 172.21.15.12:0/3361624524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c00ca470 con 0x7f65c00d15c0 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65c4e70640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c015e7a0 0x7f65c00b6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65b7fff640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c00c9a00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f65a8000c00 tx=0x7f65a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.881+0000 7f65bd7fa640 1 -- 172.21.15.12:0/3361624524 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8032880 con 0x7f65c00d15c0 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.882+0000 7f65bd7fa640 1 -- 172.21.15.12:0/3361624524 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a8032a20 con 0x7f65c00d15c0 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.882+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c00ca750 con 0x7f65c00d15c0 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.882+0000 7f65bd7fa640 1 -- 172.21.15.12:0/3361624524 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8032e10 con 0x7f65c00d15c0 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.882+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c00cad10 con 0x7f65c00d15c0 2024-04-15T15:22:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.883+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c00689e0 con 0x7f65c00d15c0 2024-04-15T15:22:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.883+0000 7f65bd7fa640 1 -- 172.21.15.12:0/3361624524 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f65a80453f0 con 0x7f65c00d15c0 2024-04-15T15:22:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.883+0000 7f65bd7fa640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f65a006c8f0 0x7f65a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.884+0000 7f65bd7fa640 1 -- 172.21.15.12:0/3361624524 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f65a80adf10 con 0x7f65c00d15c0 2024-04-15T15:22:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.884+0000 7f65bffff640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f65a006c8f0 0x7f65a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.884+0000 7f65bffff640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f65a006c8f0 0x7f65a006edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f65b0007fd0 tx=0x7f65b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:08.888+0000 7f65bd7fa640 1 -- 172.21.15.12:0/3361624524 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65a8040070 con 0x7f65c00d15c0 2024-04-15T15:22:09.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.072+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f65c00d1d60 con 0x7f65c00d15c0 2024-04-15T15:22:09.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.073+0000 7f65bd7fa640 1 -- 172.21.15.12:0/3361624524 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f65a807c0b0 con 0x7f65c00d15c0 2024-04-15T15:22:09.074 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f65a006c8f0 msgr2=0x7f65a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f65a006c8f0 0x7f65a006edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f65b0007fd0 tx=0x7f65b0009040 comp rx=0 tx=0).stop 2024-04-15T15:22:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 msgr2=0x7f65c00c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c00c9a00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f65a8000c00 tx=0x7f65a80046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 shutdown_connections 2024-04-15T15:22:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f65a006c8f0 0x7f65a006edb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c015e7a0 0x7f65c00b6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65c00d15c0 0x7f65c00c9a00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 --2- 172.21.15.12:0/3361624524 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f65c006d050 0x7f65c014e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 >> 172.21.15.12:0/3361624524 conn(0x7f65c008bf60 msgr2=0x7f65c00b4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 shutdown_connections 2024-04-15T15:22:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.075+0000 7f65c68d3640 1 -- 172.21.15.12:0/3361624524 wait complete. 2024-04-15T15:22:09.087 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.191+0000 7f71ac331640 1 Processor -- start 2024-04-15T15:22:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.191+0000 7f71ac331640 1 -- start start 2024-04-15T15:22:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71ac331640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a415e700 0x7f71a40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71ac331640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f71a40d2360 0x7f71a40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71ac331640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f71a415eae0 0x7f71a40d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71ac331640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f71a40b53b0 con 0x7f71a415eae0 2024-04-15T15:22:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71ac331640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f71a40b47a0 con 0x7f71a40d2360 2024-04-15T15:22:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71ac331640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f71a40b4a80 con 0x7f71a415e700 2024-04-15T15:22:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa8ce640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f71a415eae0 0x7f71a40d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa8ce640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f71a415eae0 0x7f71a40d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53060/0 (socket says 172.21.15.12:53060) 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa8ce640 1 -- 172.21.15.12:0/179697011 learned_addr learned my addr 172.21.15.12:0/179697011 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71a98cc640 1 --2- 172.21.15.12:0/179697011 >> v2:172.21.15.55:3300/0 conn(0x7f71a40d2360 0x7f71a40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa8ce640 1 -- 172.21.15.12:0/179697011 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a415e700 msgr2=0x7f71a40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa0cd640 1 --2- 172.21.15.12:0/179697011 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a415e700 0x7f71a40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa8ce640 1 --2- 172.21.15.12:0/179697011 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a415e700 0x7f71a40b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa8ce640 1 -- 172.21.15.12:0/179697011 >> v2:172.21.15.55:3300/0 conn(0x7f71a40d2360 msgr2=0x7f71a40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa8ce640 1 --2- 172.21.15.12:0/179697011 >> v2:172.21.15.55:3300/0 conn(0x7f71a40d2360 0x7f71a40d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.192+0000 7f71aa8ce640 1 -- 172.21.15.12:0/179697011 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a40b4d60 con 0x7f71a415eae0 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.193+0000 7f71aa8ce640 1 --2- 172.21.15.12:0/179697011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40d30a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f71a40d2100 tx=0x7f719c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfef92b113540de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.193+0000 7f71a90cb640 1 -- 172.21.15.12:0/179697011 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c035020 con 0x7f71a415eae0 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.193+0000 7f71a90cb640 1 -- 172.21.15.12:0/179697011 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719c031870 con 0x7f71a415eae0 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.193+0000 7f71a90cb640 1 -- 172.21.15.12:0/179697011 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c031c40 con 0x7f71a415eae0 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 -- 172.21.15.12:0/179697011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 msgr2=0x7f71a40d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 --2- 172.21.15.12:0/179697011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40d30a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f71a40d2100 tx=0x7f719c02b910 comp rx=0 tx=0).stop 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 -- 172.21.15.12:0/179697011 shutdown_connections 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 --2- 172.21.15.12:0/179697011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40d30a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 --2- 172.21.15.12:0/179697011 >> v2:172.21.15.55:3300/0 conn(0x7f71a40d2360 0x7f71a40d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 --2- 172.21.15.12:0/179697011 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a415e700 0x7f71a40b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 -- 172.21.15.12:0/179697011 >> 172.21.15.12:0/179697011 conn(0x7f71a408bf60 msgr2=0x7f71a40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 -- 172.21.15.12:0/179697011 shutdown_connections 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.194+0000 7f71ac331640 1 -- 172.21.15.12:0/179697011 wait complete. 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.195+0000 7f71ac331640 1 Processor -- start 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.195+0000 7f71ac331640 1 -- start start 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71ac331640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a40d2360 0x7f71a406dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71ac331640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71a415e700 0x7f71a40c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71ac331640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71ac331640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40b5d80 con 0x7f71a415eae0 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71ac331640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40b4450 con 0x7f71a415e700 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71ac331640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f71a40d1d40 con 0x7f71a40d2360 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa8ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa8ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53068/0 (socket says 172.21.15.12:53068) 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa8ce640 1 -- 172.21.15.12:0/774183964 learned_addr learned my addr 172.21.15.12:0/774183964 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71a98cc640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71a415e700 0x7f71a40c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa0cd640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a40d2360 0x7f71a406dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa8ce640 1 -- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a40d2360 msgr2=0x7f71a406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa8ce640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a40d2360 0x7f71a406dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa8ce640 1 -- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71a415e700 msgr2=0x7f71a40c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa8ce640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71a415e700 0x7f71a40c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.196+0000 7f71aa8ce640 1 -- 172.21.15.12:0/774183964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a40b7180 con 0x7f71a415eae0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.197+0000 7f71aa8ce640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40c8720 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f719c02b950 tx=0x7f719c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.197+0000 7f719affd640 1 -- 172.21.15.12:0/774183964 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c040070 con 0x7f71a415eae0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.197+0000 7f719affd640 1 -- 172.21.15.12:0/774183964 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719c03a420 con 0x7f71a415eae0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.197+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a414e840 con 0x7f71a415eae0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.197+0000 7f719affd640 1 -- 172.21.15.12:0/774183964 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c035040 con 0x7f71a415eae0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.197+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a414bf70 con 0x7f71a415eae0 2024-04-15T15:22:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.198+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a40689e0 con 0x7f71a415eae0 2024-04-15T15:22:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.198+0000 7f719affd640 1 -- 172.21.15.12:0/774183964 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f719c038020 con 0x7f71a415eae0 2024-04-15T15:22:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.199+0000 7f719affd640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f718806c8f0 0x7f718806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.199+0000 7f719affd640 1 -- 172.21.15.12:0/774183964 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f719c0ade50 con 0x7f71a415eae0 2024-04-15T15:22:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.199+0000 7f71aa0cd640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f718806c8f0 0x7f718806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.199+0000 7f71aa0cd640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f718806c8f0 0x7f718806edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f71a000f560 tx=0x7f71a0007800 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.203+0000 7f719affd640 1 -- 172.21.15.12:0/774183964 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f71a40b7180 con 0x7f71a415eae0 2024-04-15T15:22:09.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.400+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f71a40ba500 con 0x7f71a415eae0 2024-04-15T15:22:09.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.401+0000 7f719affd640 1 -- 172.21.15.12:0/774183964 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f719c07b880 con 0x7f71a415eae0 2024-04-15T15:22:09.402 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:09.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f718806c8f0 msgr2=0x7f718806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f718806c8f0 0x7f718806edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f71a000f560 tx=0x7f71a0007800 comp rx=0 tx=0).stop 2024-04-15T15:22:09.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 msgr2=0x7f71a40c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40c8720 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f719c02b950 tx=0x7f719c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 shutdown_connections 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71a415eae0 0x7f71a40c8720 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f718806c8f0 0x7f718806edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71a415e700 0x7f71a40c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 --2- 172.21.15.12:0/774183964 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f71a40d2360 0x7f71a406dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 >> 172.21.15.12:0/774183964 conn(0x7f71a408bf60 msgr2=0x7f71a40b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 shutdown_connections 2024-04-15T15:22:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.403+0000 7f71ac331640 1 -- 172.21.15.12:0/774183964 wait complete. 2024-04-15T15:22:09.415 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi055.front.sepia.ceph.com... 2024-04-15T15:22:09.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:09.416 DEBUG:teuthology.orchestra.run.smithi055:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:22:09.451 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:09.451 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:22:09.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:09.452 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:22:09.516 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:22:09.517 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:09.533 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:22:09.533 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:09.589 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.733+0000 7f9fcd11c640 1 Processor -- start 2024-04-15T15:22:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.733+0000 7f9fcd11c640 1 -- start start 2024-04-15T15:22:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fcd11c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9fc80cff10 0x7f9fc80b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fcd11c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fc80b1f40 con 0x7f9fc80b1480 2024-04-15T15:22:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fcd11c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fc80cda30 con 0x7f9fc815fda0 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fcd11c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9fc80cdd10 con 0x7f9fc80cff10 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fbe59c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9fc80b1480 0x7f9fc80cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34740/0 (socket says 172.21.15.12:34740) 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fbe59c640 1 -- 172.21.15.12:0/2347976947 learned_addr learned my addr 172.21.15.12:0/2347976947 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc759e640 1 --2- 172.21.15.12:0/2347976947 >> v2:172.21.15.55:3300/0 conn(0x7f9fc80cff10 0x7f9fc80b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 244912179 0 0) 0x7f9fc80b1f40 con 0x7f9fc80b1480 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fb4003520 con 0x7f9fc80b1480 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2746791772 0 0) 0x7f9fb4003520 con 0x7f9fc80b1480 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 >> v2:172.21.15.55:3300/0 conn(0x7f9fc80cff10 msgr2=0x7f9fc80b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 --2- 172.21.15.12:0/2347976947 >> v2:172.21.15.55:3300/0 conn(0x7f9fc80cff10 0x7f9fc80b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 >> v1:172.21.15.55:6789/0 conn(0x7f9fc815fda0 legacy=0x7f9fc80ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fc80cdff0 con 0x7f9fc80b1480 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f9fb0001670 con 0x7f9fc80b1480 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 >> v1:172.21.15.12:6789/0 conn(0x7f9fc80b1480 legacy=0x7f9fc80cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40039b0 0x7f9fb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fb4004800 0x7f9fb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fb40051c0 0x7f9fb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc80cdd10 con 0x7f9fb40051c0 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb400c060 con 0x7f9fb4004800 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.734+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb400c260 con 0x7f9fb40039b0 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc759e640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fb40051c0 0x7f9fb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fbe59c640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fb4004800 0x7f9fb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc6d9d640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40039b0 0x7f9fb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc759e640 1 -- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40039b0 msgr2=0x7f9fb4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc759e640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40039b0 0x7f9fb4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc759e640 1 -- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fb4004800 msgr2=0x7f9fb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc759e640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fb4004800 0x7f9fb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc759e640 1 -- 172.21.15.12:0/2347976947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb4003520 con 0x7f9fb40051c0 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc6d9d640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40039b0 0x7f9fb4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.735+0000 7f9fc759e640 1 -- 172.21.15.12:0/2347976947 reap_dead start 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fc759e640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fb40051c0 0x7f9fb400b9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0009fc0 tx=0x7f9fb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5d66d5a8a156c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0009c10 con 0x7f9fb40051c0 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc002cca0 con 0x7f9fb40051c0 2024-04-15T15:22:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fc659c640 1 -- 172.21.15.12:0/2347976947 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0003bd0 con 0x7f9fb40051c0 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fcd11c640 1 -- 172.21.15.12:0/2347976947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fb40051c0 msgr2=0x7f9fb400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fcd11c640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fb40051c0 0x7f9fb400b9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0009fc0 tx=0x7f9fb4005830 comp rx=0 tx=0).stop 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fcd11c640 1 -- 172.21.15.12:0/2347976947 shutdown_connections 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fcd11c640 1 --2- 172.21.15.12:0/2347976947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fb40051c0 0x7f9fb400b9c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fcd11c640 1 -- 172.21.15.12:0/2347976947 >> 172.21.15.12:0/2347976947 conn(0x7f9fc80b6870 msgr2=0x7f9fc80b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fcd11c640 1 -- 172.21.15.12:0/2347976947 shutdown_connections 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.736+0000 7f9fcd11c640 1 -- 172.21.15.12:0/2347976947 wait complete. 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.738+0000 7f9fcd11c640 1 Processor -- start 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.738+0000 7f9fcd11c640 1 -- start start 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.738+0000 7f9fcd11c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40051c0 0x7f9fc809b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.738+0000 7f9fcd11c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fc81516b0 0x7f9fc8152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.738+0000 7f9fcd11c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fc8150e30 0x7f9fc815bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.738+0000 7f9fcd11c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc80b0790 con 0x7f9fc8150e30 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.738+0000 7f9fcd11c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc8160180 con 0x7f9fc81516b0 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.738+0000 7f9fcd11c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9fc80b18a0 con 0x7f9fb40051c0 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fc8150e30 0x7f9fc815bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fc8150e30 0x7f9fc815bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53086/0 (socket says 172.21.15.12:53086) 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 -- 172.21.15.12:0/4214109287 learned_addr learned my addr 172.21.15.12:0/4214109287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc6d9d640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40051c0 0x7f9fc809b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fbe59c640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fc81516b0 0x7f9fc8152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 -- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40051c0 msgr2=0x7f9fc809b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40051c0 0x7f9fc809b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 -- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fc81516b0 msgr2=0x7f9fc8152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fc81516b0 0x7f9fc8152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 -- 172.21.15.12:0/4214109287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fc80b44f0 con 0x7f9fc8150e30 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fbe59c640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fc81516b0 0x7f9fc8152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fc759e640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fc8150e30 0x7f9fc815bcb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9fc002c6d0 tx=0x7f9fc0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fbffff640 1 -- 172.21.15.12:0/4214109287 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0042020 con 0x7f9fc8150e30 2024-04-15T15:22:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fbffff640 1 -- 172.21.15.12:0/4214109287 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc002ce50 con 0x7f9fc8150e30 2024-04-15T15:22:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fc80b4790 con 0x7f9fc8150e30 2024-04-15T15:22:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fbffff640 1 -- 172.21.15.12:0/4214109287 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0003b90 con 0x7f9fc8150e30 2024-04-15T15:22:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.739+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fc81527b0 con 0x7f9fc8150e30 2024-04-15T15:22:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.740+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fc80d1430 con 0x7f9fc8150e30 2024-04-15T15:22:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.741+0000 7f9fbffff640 1 -- 172.21.15.12:0/4214109287 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9fc0045020 con 0x7f9fc8150e30 2024-04-15T15:22:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.741+0000 7f9fbffff640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fa806c8f0 0x7f9fa806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.742+0000 7f9fbffff640 1 -- 172.21.15.12:0/4214109287 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f9fc00ae8d0 con 0x7f9fc8150e30 2024-04-15T15:22:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.742+0000 7f9fc6d9d640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fa806c8f0 0x7f9fa806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.742+0000 7f9fc6d9d640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fa806c8f0 0x7f9fa806edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8000c00 tx=0x7f9fb8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.746+0000 7f9fbffff640 1 -- 172.21.15.12:0/4214109287 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fc80d1430 con 0x7f9fc8150e30 2024-04-15T15:22:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.935+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9fc80689e0 con 0x7f9fc8150e30 2024-04-15T15:22:09.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.936+0000 7f9fbffff640 1 -- 172.21.15.12:0/4214109287 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f9fc004c020 con 0x7f9fc8150e30 2024-04-15T15:22:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:09.938 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fa806c8f0 msgr2=0x7f9fa806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fa806c8f0 0x7f9fa806edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8000c00 tx=0x7f9fb8009040 comp rx=0 tx=0).stop 2024-04-15T15:22:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fc8150e30 msgr2=0x7f9fc815bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fc8150e30 0x7f9fc815bcb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9fc002c6d0 tx=0x7f9fc0009750 comp rx=0 tx=0).stop 2024-04-15T15:22:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 shutdown_connections 2024-04-15T15:22:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fc8150e30 0x7f9fc815bcb0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fa806c8f0 0x7f9fa806edb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fc81516b0 0x7f9fc8152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 --2- 172.21.15.12:0/4214109287 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fb40051c0 0x7f9fc809b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.938+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 >> 172.21.15.12:0/4214109287 conn(0x7f9fc80b6870 msgr2=0x7f9fc80ce640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.939+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 shutdown_connections 2024-04-15T15:22:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:09.939+0000 7f9fcd11c640 1 -- 172.21.15.12:0/4214109287 wait complete. 2024-04-15T15:22:09.951 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:09.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:10.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.058+0000 7f870edbd640 1 Processor -- start 2024-04-15T15:22:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.058+0000 7f870edbd640 1 -- start start 2024-04-15T15:22:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.058+0000 7f870edbd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f870815c8d0 0x7f87080cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.058+0000 7f870edbd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f870809b650 0x7f870809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.058+0000 7f870edbd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87080d0350 con 0x7f870809af10 2024-04-15T15:22:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.058+0000 7f870edbd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f87080b5b00 con 0x7f870809b650 2024-04-15T15:22:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.058+0000 7f870edbd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f870809b340 con 0x7f870815c8d0 2024-04-15T15:22:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f8707fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f870809b650 0x7f870809a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f870d35a640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f870809af10 0x7f87080d0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34744/0 (socket says 172.21.15.12:34744) 2024-04-15T15:22:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f870d35a640 1 -- 172.21.15.12:0/2156478063 learned_addr learned my addr 172.21.15.12:0/2156478063 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f8707fff640 1 -- 172.21.15.12:0/2156478063 >> v2:172.21.15.55:3300/0 conn(0x7f870815c8d0 msgr2=0x7f87080cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f870cb59640 1 --2- 172.21.15.12:0/2156478063 >> v2:172.21.15.55:3300/0 conn(0x7f870815c8d0 0x7f87080cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f8707fff640 1 --2- 172.21.15.12:0/2156478063 >> v2:172.21.15.55:3300/0 conn(0x7f870815c8d0 0x7f87080cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f87077fe640 1 -- 172.21.15.12:0/2156478063 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103450026 0 0) 0x7f87080d0350 con 0x7f870809af10 2024-04-15T15:22:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f8707fff640 1 -- 172.21.15.12:0/2156478063 >> v1:172.21.15.12:6789/0 conn(0x7f870809af10 legacy=0x7f87080d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.059+0000 7f8707fff640 1 -- 172.21.15.12:0/2156478063 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87080ce100 con 0x7f870809b650 2024-04-15T15:22:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f8707fff640 1 --2- 172.21.15.12:0/2156478063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f870809b650 0x7f870809a9d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f870809b620 tx=0x7f86f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b51ff97fb8a609 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f87077fe640 1 -- 172.21.15.12:0/2156478063 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f40315a0 con 0x7f870809b650 2024-04-15T15:22:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f87077fe640 1 -- 172.21.15.12:0/2156478063 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f4037070 con 0x7f870809b650 2024-04-15T15:22:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f87077fe640 1 -- 172.21.15.12:0/2156478063 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4031960 con 0x7f870809b650 2024-04-15T15:22:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f870edbd640 1 -- 172.21.15.12:0/2156478063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f870809b650 msgr2=0x7f870809a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f870edbd640 1 --2- 172.21.15.12:0/2156478063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f870809b650 0x7f870809a9d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f870809b620 tx=0x7f86f402b840 comp rx=0 tx=0).stop 2024-04-15T15:22:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f870edbd640 1 -- 172.21.15.12:0/2156478063 shutdown_connections 2024-04-15T15:22:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f870edbd640 1 --2- 172.21.15.12:0/2156478063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f870809b650 0x7f870809a9d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f870edbd640 1 --2- 172.21.15.12:0/2156478063 >> v2:172.21.15.55:3300/0 conn(0x7f870815c8d0 0x7f87080cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f870edbd640 1 -- 172.21.15.12:0/2156478063 >> 172.21.15.12:0/2156478063 conn(0x7f87080b3fb0 msgr2=0x7f87080b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f870edbd640 1 -- 172.21.15.12:0/2156478063 shutdown_connections 2024-04-15T15:22:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.060+0000 7f870edbd640 1 -- 172.21.15.12:0/2156478063 wait complete. 2024-04-15T15:22:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.062+0000 7f870edbd640 1 Processor -- start 2024-04-15T15:22:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.062+0000 7f870edbd640 1 -- start start 2024-04-15T15:22:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.062+0000 7f870edbd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f870809af10 0x7f87080c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.062+0000 7f870edbd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f870809b650 0x7f870809cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.062+0000 7f870edbd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87080c5ff0 0x7f870815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870edbd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f87080b6fa0 con 0x7f87080c5ff0 2024-04-15T15:22:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870edbd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f87080b5790 con 0x7f870809af10 2024-04-15T15:22:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870edbd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f87080b7120 con 0x7f870809b650 2024-04-15T15:22:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87080c5ff0 0x7f870815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87080c5ff0 0x7f870815aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53098/0 (socket says 172.21.15.12:53098) 2024-04-15T15:22:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 -- 172.21.15.12:0/3363290332 learned_addr learned my addr 172.21.15.12:0/3363290332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870cb59640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f870809af10 0x7f87080c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f8707fff640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f870809b650 0x7f870809cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 -- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f870809b650 msgr2=0x7f870809cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f870809b650 0x7f870809cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 -- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f870809af10 msgr2=0x7f87080c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f870809af10 0x7f87080c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 -- 172.21.15.12:0/3363290332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87080c6fc0 con 0x7f87080c5ff0 2024-04-15T15:22:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870cb59640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f870809af10 0x7f87080c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870d35a640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87080c5ff0 0x7f870815aa20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f870000c700 tx=0x7f870000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f87057fa640 1 -- 172.21.15.12:0/3363290332 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700018070 con 0x7f87080c5ff0 2024-04-15T15:22:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f87057fa640 1 -- 172.21.15.12:0/3363290332 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870000ce10 con 0x7f87080c5ff0 2024-04-15T15:22:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f87057fa640 1 -- 172.21.15.12:0/3363290332 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870000a860 con 0x7f87080c5ff0 2024-04-15T15:22:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87080c6a00 con 0x7f87080c5ff0 2024-04-15T15:22:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.063+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f870809ecc0 con 0x7f87080c5ff0 2024-04-15T15:22:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.064+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f870809ead0 con 0x7f87080c5ff0 2024-04-15T15:22:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.065+0000 7f87057fa640 1 -- 172.21.15.12:0/3363290332 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f870001b070 con 0x7f87080c5ff0 2024-04-15T15:22:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.065+0000 7f87057fa640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f86ec06c8f0 0x7f86ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.065+0000 7f87057fa640 1 -- 172.21.15.12:0/3363290332 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f870008b500 con 0x7f87080c5ff0 2024-04-15T15:22:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.065+0000 7f870cb59640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f86ec06c8f0 0x7f86ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.066+0000 7f870cb59640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f86ec06c8f0 0x7f86ec06edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f86fc00a9d0 tx=0x7f86fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.070+0000 7f87057fa640 1 -- 172.21.15.12:0/3363290332 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87000594f0 con 0x7f87080c5ff0 2024-04-15T15:22:10.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.252+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f87080690f0 con 0x7f87080c5ff0 2024-04-15T15:22:10.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.253+0000 7f87057fa640 1 -- 172.21.15.12:0/3363290332 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f8700058df0 con 0x7f87080c5ff0 2024-04-15T15:22:10.254 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:10.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f86ec06c8f0 msgr2=0x7f86ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f86ec06c8f0 0x7f86ec06edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f86fc00a9d0 tx=0x7f86fc008040 comp rx=0 tx=0).stop 2024-04-15T15:22:10.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87080c5ff0 msgr2=0x7f870815aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87080c5ff0 0x7f870815aa20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f870000c700 tx=0x7f870000cbc0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 shutdown_connections 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87080c5ff0 0x7f870815aa20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f870809b650 0x7f870809cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f86ec06c8f0 0x7f86ec06edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 --2- 172.21.15.12:0/3363290332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f870809af10 0x7f87080c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 >> 172.21.15.12:0/3363290332 conn(0x7f87080b3fb0 msgr2=0x7f8708070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 shutdown_connections 2024-04-15T15:22:10.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.255+0000 7f870edbd640 1 -- 172.21.15.12:0/3363290332 wait complete. 2024-04-15T15:22:10.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:10.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.374+0000 7fa7ce5f4640 1 Processor -- start 2024-04-15T15:22:10.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.374+0000 7fa7ce5f4640 1 -- start start 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7ce5f4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 0x7fa7c80d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7ce5f4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa7c80d1440 0x7fa7c80b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7ce5f4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c80d2480 con 0x7fa7c815e790 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7ce5f4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa7c80d2db0 con 0x7fa7c80d1440 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7ce5f4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa7c80d2760 con 0x7fa7c80d3240 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7c7fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa7c815e790 0x7fa7c80cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34750/0 (socket says 172.21.15.12:34750) 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7c7fff640 1 -- 172.21.15.12:0/3008651476 learned_addr learned my addr 172.21.15.12:0/3008651476 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731520601 0 0) 0x7fa7c80d2480 con 0x7fa7c815e790 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7b0003520 con 0x7fa7c815e790 2024-04-15T15:22:10.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7c77fe640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 0x7fa7c80d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7ccb91640 1 --2- 172.21.15.12:0/3008651476 >> v2:172.21.15.55:3300/0 conn(0x7fa7c80d1440 0x7fa7c80b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.375+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 478255510 0 0) 0x7fa7b0003520 con 0x7fa7c815e790 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 msgr2=0x7fa7c80d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 0x7fa7c80d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 >> v2:172.21.15.55:3300/0 conn(0x7fa7c80d1440 msgr2=0x7fa7c80b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 --2- 172.21.15.12:0/3008651476 >> v2:172.21.15.55:3300/0 conn(0x7fa7c80d1440 0x7fa7c80b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c80d2a40 con 0x7fa7c815e790 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fa7b8001670 con 0x7fa7c815e790 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 >> v1:172.21.15.12:6789/0 conn(0x7fa7c815e790 legacy=0x7fa7c80cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 0x7fa7b00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b0003f30 0x7fa7b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0004d40 0x7fa7b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80d2db0 con 0x7fa7b0003f30 2024-04-15T15:22:10.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80d2760 con 0x7fa7b0004d40 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b000b9c0 con 0x7fa7c80d3240 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c77fe640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b0003f30 0x7fa7b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c7fff640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 0x7fa7b00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7ccb91640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0004d40 0x7fa7b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c77fe640 1 -- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 msgr2=0x7fa7b00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c77fe640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 0x7fa7b00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c77fe640 1 -- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0004d40 msgr2=0x7fa7b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c77fe640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0004d40 0x7fa7b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c77fe640 1 -- 172.21.15.12:0/3008651476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b0003520 con 0x7fa7b0003f30 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7c7fff640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80d3240 0x7fa7b00039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.376+0000 7fa7ccb91640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0004d40 0x7fa7b00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7c77fe640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b0003f30 0x7fa7b0004800 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009ac0 tx=0x7fa7b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34dbae06318e5ae6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc039070 con 0x7fa7b0003f30 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7bc02c6b0 con 0x7fa7b0003f30 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7c6ffd640 1 -- 172.21.15.12:0/3008651476 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc009690 con 0x7fa7b0003f30 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3008651476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b0003f30 msgr2=0x7fa7b0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7ce5f4640 1 --2- 172.21.15.12:0/3008651476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7b0003f30 0x7fa7b0004800 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009ac0 tx=0x7fa7b0005830 comp rx=0 tx=0).stop 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7ccb91640 1 -- 172.21.15.12:0/3008651476 reap_dead start 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3008651476 shutdown_connections 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3008651476 >> 172.21.15.12:0/3008651476 conn(0x7fa7c808bf60 msgr2=0x7fa7c80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.377+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3008651476 shutdown_connections 2024-04-15T15:22:10.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.378+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3008651476 wait complete. 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.379+0000 7fa7ce5f4640 1 Processor -- start 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.379+0000 7fa7ce5f4640 1 -- start start 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ce5f4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0003f30 0x7fa7c80cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ce5f4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80c8e00 0x7fa7c80b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ce5f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c815d7c0 0x7fa7c815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ce5f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80d5180 con 0x7fa7c815d7c0 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ce5f4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80d0430 con 0x7fa7b0003f30 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ce5f4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa7c80d5300 con 0x7fa7c80c8e00 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c815d7c0 0x7fa7c815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c815d7c0 0x7fa7c815dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53122/0 (socket says 172.21.15.12:53122) 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 -- 172.21.15.12:0/3449424893 learned_addr learned my addr 172.21.15.12:0/3449424893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 -- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80c8e00 msgr2=0x7fa7c80b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80c8e00 0x7fa7c80b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 -- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0003f30 msgr2=0x7fa7c80cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7c7fff640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0003f30 0x7fa7c80cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0003f30 0x7fa7c80cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 -- 172.21.15.12:0/3449424893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c815e180 con 0x7fa7c815d7c0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ccb91640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c815d7c0 0x7fa7c815dc40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80b7290 tx=0x7fa7c0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7c4ff9640 1 -- 172.21.15.12:0/3449424893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0010d80 con 0x7fa7c815d7c0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7c4ff9640 1 -- 172.21.15.12:0/3449424893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c0003990 con 0x7fa7c815d7c0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c80c9630 con 0x7fa7c815d7c0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7c4ff9640 1 -- 172.21.15.12:0/3449424893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0003d80 con 0x7fa7c815d7c0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.380+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c80c9bf0 con 0x7fa7c815d7c0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.381+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7c80689e0 con 0x7fa7c815d7c0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.382+0000 7fa7c4ff9640 1 -- 172.21.15.12:0/3449424893 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa7c001e030 con 0x7fa7c815d7c0 2024-04-15T15:22:10.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.382+0000 7fa7c4ff9640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa7b406c8f0 0x7fa7b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.382+0000 7fa7c4ff9640 1 -- 172.21.15.12:0/3449424893 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fa7c008c790 con 0x7fa7c815d7c0 2024-04-15T15:22:10.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.383+0000 7fa7c7fff640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa7b406c8f0 0x7fa7b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.383+0000 7fa7c7fff640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa7b406c8f0 0x7fa7b406edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8001ce0 tx=0x7fa7b8001c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.387+0000 7fa7c4ff9640 1 -- 172.21.15.12:0/3449424893 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa7c0025020 con 0x7fa7c815d7c0 2024-04-15T15:22:10.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.569+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa7c808e720 con 0x7fa7c815d7c0 2024-04-15T15:22:10.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.569+0000 7fa7c4ff9640 1 -- 172.21.15.12:0/3449424893 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa7c815e180 con 0x7fa7c815d7c0 2024-04-15T15:22:10.571 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:10.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.571+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa7b406c8f0 msgr2=0x7fa7b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.571+0000 7fa7ce5f4640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa7b406c8f0 0x7fa7b406edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8001ce0 tx=0x7fa7b8001c50 comp rx=0 tx=0).stop 2024-04-15T15:22:10.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.571+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c815d7c0 msgr2=0x7fa7c815dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.571+0000 7fa7ce5f4640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c815d7c0 0x7fa7c815dc40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80b7290 tx=0x7fa7c0010590 comp rx=0 tx=0).stop 2024-04-15T15:22:10.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.572+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 shutdown_connections 2024-04-15T15:22:10.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.572+0000 7fa7ce5f4640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7c815d7c0 0x7fa7c815dc40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.572+0000 7fa7ce5f4640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7c80c8e00 0x7fa7c80b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.572+0000 7fa7ce5f4640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa7b406c8f0 0x7fa7b406edb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.572+0000 7fa7ce5f4640 1 --2- 172.21.15.12:0/3449424893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7b0003f30 0x7fa7c80cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.572+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 >> 172.21.15.12:0/3449424893 conn(0x7fa7c808bf60 msgr2=0x7fa7c806e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:10.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.572+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 shutdown_connections 2024-04-15T15:22:10.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.572+0000 7fa7ce5f4640 1 -- 172.21.15.12:0/3449424893 wait complete. 2024-04-15T15:22:10.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:10.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.691+0000 7f6942115640 1 Processor -- start 2024-04-15T15:22:10.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.691+0000 7f6942115640 1 -- start start 2024-04-15T15:22:10.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f6942115640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f693c0ce8e0 0x7f693c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f6942115640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f693c09b0e0 0x7f693c0cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f6942115640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f693c06ff30 con 0x7f693c071d90 2024-04-15T15:22:10.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f6942115640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f693c070820 con 0x7f693c09b0e0 2024-04-15T15:22:10.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f6942115640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f693c0cecc0 con 0x7f693c0ce8e0 2024-04-15T15:22:10.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f693bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f693c09b0e0 0x7f693c0cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f693bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f693c09b0e0 0x7f693c0cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53132/0 (socket says 172.21.15.12:53132) 2024-04-15T15:22:10.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f693bfff640 1 -- 172.21.15.12:0/2278107309 learned_addr learned my addr 172.21.15.12:0/2278107309 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:10.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f693b7fe640 1 --2- 172.21.15.12:0/2278107309 >> v2:172.21.15.55:3300/0 conn(0x7f693c0ce8e0 0x7f693c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f693bfff640 1 -- 172.21.15.12:0/2278107309 >> v2:172.21.15.55:3300/0 conn(0x7f693c0ce8e0 msgr2=0x7f693c070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:10.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f693bfff640 1 --2- 172.21.15.12:0/2278107309 >> v2:172.21.15.55:3300/0 conn(0x7f693c0ce8e0 0x7f693c070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f693bfff640 1 -- 172.21.15.12:0/2278107309 >> v1:172.21.15.55:6789/0 conn(0x7f693c071d90 legacy=0x7f693c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.692+0000 7f693bfff640 1 -- 172.21.15.12:0/2278107309 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693c09b9d0 con 0x7f693c09b0e0 2024-04-15T15:22:10.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.693+0000 7f693bfff640 1 --2- 172.21.15.12:0/2278107309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c09b0e0 0x7f693c0cce80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f693c0732e0 tx=0x7f693402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1d4f549e14f355 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.693+0000 7f693a7fc640 1 -- 172.21.15.12:0/2278107309 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693402ca80 con 0x7f693c09b0e0 2024-04-15T15:22:10.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.693+0000 7f693a7fc640 1 -- 172.21.15.12:0/2278107309 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693402cc20 con 0x7f693c09b0e0 2024-04-15T15:22:10.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.693+0000 7f693a7fc640 1 -- 172.21.15.12:0/2278107309 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69340343f0 con 0x7f693c09b0e0 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.693+0000 7f6942115640 1 -- 172.21.15.12:0/2278107309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c09b0e0 msgr2=0x7f693c0cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.693+0000 7f6942115640 1 --2- 172.21.15.12:0/2278107309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c09b0e0 0x7f693c0cce80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f693c0732e0 tx=0x7f693402bee0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.694+0000 7f6942115640 1 -- 172.21.15.12:0/2278107309 shutdown_connections 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.694+0000 7f6942115640 1 --2- 172.21.15.12:0/2278107309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c09b0e0 0x7f693c0cce80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.694+0000 7f6942115640 1 --2- 172.21.15.12:0/2278107309 >> v2:172.21.15.55:3300/0 conn(0x7f693c0ce8e0 0x7f693c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.694+0000 7f6942115640 1 -- 172.21.15.12:0/2278107309 >> 172.21.15.12:0/2278107309 conn(0x7f693c0b3f40 msgr2=0x7f693c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.694+0000 7f6942115640 1 -- 172.21.15.12:0/2278107309 shutdown_connections 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.694+0000 7f6942115640 1 -- 172.21.15.12:0/2278107309 wait complete. 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.695+0000 7f6942115640 1 Processor -- start 2024-04-15T15:22:10.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.695+0000 7f6942115640 1 -- start start 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f6942115640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f693c071d90 0x7f693c1ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f6942115640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f693c09b0e0 0x7f693c0cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f6942115640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c0ce8e0 0x7f693c0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f6942115640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f693c0704b0 con 0x7f693c0ce8e0 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f6942115640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f693c0701d0 con 0x7f693c09b0e0 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f6942115640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f693c071970 con 0x7f693c071d90 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c0ce8e0 0x7f693c0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c0ce8e0 0x7f693c0cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53144/0 (socket says 172.21.15.12:53144) 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 -- 172.21.15.12:0/3603849476 learned_addr learned my addr 172.21.15.12:0/3603849476 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693affd640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f693c09b0e0 0x7f693c0cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693b7fe640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f693c071d90 0x7f693c1ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 -- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f693c071d90 msgr2=0x7f693c1ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f693c071d90 0x7f693c1ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 -- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f693c09b0e0 msgr2=0x7f693c0cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f693c09b0e0 0x7f693c0cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 -- 172.21.15.12:0/3603849476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693c0b5d50 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693affd640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f693c09b0e0 0x7f693c0cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f693bfff640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c0ce8e0 0x7f693c0cba80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6934036040 tx=0x7f6934004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.696+0000 7f6923fff640 1 -- 172.21.15.12:0/3603849476 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6934043020 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.697+0000 7f6923fff640 1 -- 172.21.15.12:0/3603849476 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693403e450 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.697+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f693c0b5ff0 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.697+0000 7f6923fff640 1 -- 172.21.15.12:0/3603849476 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6934009510 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.697+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f693c1dfb60 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.698+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f693c0689e0 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.698+0000 7f6923fff640 1 -- 172.21.15.12:0/3603849476 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6934046020 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.698+0000 7f6923fff640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f692806c920 0x7f692806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.699+0000 7f6923fff640 1 -- 172.21.15.12:0/3603849476 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f69340ae7c0 con 0x7f693c0ce8e0 2024-04-15T15:22:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.699+0000 7f693b7fe640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f692806c920 0x7f692806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:10.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.699+0000 7f693b7fe640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f692806c920 0x7f692806ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f692c006fd0 tx=0x7f692c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:10.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.703+0000 7f6923fff640 1 -- 172.21.15.12:0/3603849476 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f693404d030 con 0x7f693c0ce8e0 2024-04-15T15:22:10.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.889+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f693c0b4ca0 con 0x7f693c0ce8e0 2024-04-15T15:22:10.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.889+0000 7f6923fff640 1 -- 172.21.15.12:0/3603849476 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f693407c7b0 con 0x7f693c0ce8e0 2024-04-15T15:22:10.891 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:10.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f692806c920 msgr2=0x7f692806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f692806c920 0x7f692806ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f692c006fd0 tx=0x7f692c008040 comp rx=0 tx=0).stop 2024-04-15T15:22:10.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c0ce8e0 msgr2=0x7f693c0cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:10.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c0ce8e0 0x7f693c0cba80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6934036040 tx=0x7f6934004a40 comp rx=0 tx=0).stop 2024-04-15T15:22:10.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 shutdown_connections 2024-04-15T15:22:10.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f693c0ce8e0 0x7f693c0cba80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f692806c920 0x7f692806ede0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f693c09b0e0 0x7f693c0cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 --2- 172.21.15.12:0/3603849476 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f693c071d90 0x7f693c1ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:10.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 >> 172.21.15.12:0/3603849476 conn(0x7f693c0b3f40 msgr2=0x7f693c0af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:10.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 shutdown_connections 2024-04-15T15:22:10.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:10.891+0000 7f6942115640 1 -- 172.21.15.12:0/3603849476 wait complete. 2024-04-15T15:22:10.904 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.011+0000 7f22c7ec1640 1 Processor -- start 2024-04-15T15:22:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.012+0000 7f22c7ec1640 1 -- start start 2024-04-15T15:22:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.012+0000 7f22c7ec1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22c015e700 0x7f22c00b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.012+0000 7f22c7ec1640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c00d2360 0x7f22c00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.012+0000 7f22c7ec1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f22c015eae0 0x7f22c00d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.012+0000 7f22c7ec1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f22c00b53b0 con 0x7f22c015e700 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.012+0000 7f22c7ec1640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f22c00b47a0 con 0x7f22c015eae0 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.012+0000 7f22c7ec1640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f22c00b4a80 con 0x7f22c00d2360 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c5c5d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22c015e700 0x7f22c00b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c5c5d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22c015e700 0x7f22c00b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53148/0 (socket says 172.21.15.12:53148) 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c5c5d640 1 -- 172.21.15.12:0/919894637 learned_addr learned my addr 172.21.15.12:0/919894637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c545c640 1 --2- 172.21.15.12:0/919894637 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c00d2360 0x7f22c00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c645e640 1 --2- 172.21.15.12:0/919894637 >> v2:172.21.15.55:3300/0 conn(0x7f22c015eae0 0x7f22c00d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c5c5d640 1 -- 172.21.15.12:0/919894637 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c00d2360 msgr2=0x7f22c00d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c5c5d640 1 --2- 172.21.15.12:0/919894637 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c00d2360 0x7f22c00d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c5c5d640 1 -- 172.21.15.12:0/919894637 >> v2:172.21.15.55:3300/0 conn(0x7f22c015eae0 msgr2=0x7f22c00d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c5c5d640 1 --2- 172.21.15.12:0/919894637 >> v2:172.21.15.55:3300/0 conn(0x7f22c015eae0 0x7f22c00d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c5c5d640 1 -- 172.21.15.12:0/919894637 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c00b4d60 con 0x7f22c015e700 2024-04-15T15:22:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.013+0000 7f22c645e640 1 --2- 172.21.15.12:0/919894637 >> v2:172.21.15.55:3300/0 conn(0x7f22c015eae0 0x7f22c00d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c5c5d640 1 --2- 172.21.15.12:0/919894637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c015e700 0x7f22c00b5840 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f22c008c780 tx=0x7f22bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab44aa14a4e4644b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c4c5b640 1 -- 172.21.15.12:0/919894637 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc02d040 con 0x7f22c015e700 2024-04-15T15:22:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c4c5b640 1 -- 172.21.15.12:0/919894637 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22bc004230 con 0x7f22c015e700 2024-04-15T15:22:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c4c5b640 1 -- 172.21.15.12:0/919894637 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc003610 con 0x7f22c015e700 2024-04-15T15:22:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c7ec1640 1 -- 172.21.15.12:0/919894637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c015e700 msgr2=0x7f22c00b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/919894637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c015e700 0x7f22c00b5840 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f22c008c780 tx=0x7f22bc02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c7ec1640 1 -- 172.21.15.12:0/919894637 shutdown_connections 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/919894637 >> v2:172.21.15.55:3300/0 conn(0x7f22c015eae0 0x7f22c00d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/919894637 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c00d2360 0x7f22c00d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/919894637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c015e700 0x7f22c00b5840 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.014+0000 7f22c7ec1640 1 -- 172.21.15.12:0/919894637 >> 172.21.15.12:0/919894637 conn(0x7f22c008bf60 msgr2=0x7f22c00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.015+0000 7f22c7ec1640 1 -- 172.21.15.12:0/919894637 shutdown_connections 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.015+0000 7f22c7ec1640 1 -- 172.21.15.12:0/919894637 wait complete. 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.016+0000 7f22c7ec1640 1 Processor -- start 2024-04-15T15:22:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.016+0000 7f22c7ec1640 1 -- start start 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.016+0000 7f22c7ec1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c00d2360 0x7f22c01f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.016+0000 7f22c7ec1640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c015e700 0x7f22c00b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.016+0000 7f22c7ec1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22c015eae0 0x7f22c00ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.016+0000 7f22c7ec1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f22c00b5d80 con 0x7f22c00d2360 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.016+0000 7f22c7ec1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f22c00b4450 con 0x7f22c015eae0 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.016+0000 7f22c7ec1640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f22c00d1d40 con 0x7f22c015e700 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c00d2360 0x7f22c01f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c00d2360 0x7f22c01f7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53156/0 (socket says 172.21.15.12:53156) 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 -- 172.21.15.12:0/2190035849 learned_addr learned my addr 172.21.15.12:0/2190035849 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c645e640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22c015eae0 0x7f22c00ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c545c640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c015e700 0x7f22c00b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 -- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c015e700 msgr2=0x7f22c00b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c015e700 0x7f22c00b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 -- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22c015eae0 msgr2=0x7f22c00ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22c015eae0 0x7f22c00ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 -- 172.21.15.12:0/2190035849 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c00cc3a0 con 0x7f22c00d2360 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c5c5d640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c00d2360 0x7f22c01f7550 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0090b0 tx=0x7f22bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22b67fc640 1 -- 172.21.15.12:0/2190035849 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc006040 con 0x7f22c00d2360 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22b67fc640 1 -- 172.21.15.12:0/2190035849 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22bc006200 con 0x7f22c00d2360 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c00ca890 con 0x7f22c00d2360 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22b67fc640 1 -- 172.21.15.12:0/2190035849 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc0065f0 con 0x7f22c00d2360 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.017+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c00cae50 con 0x7f22c00d2360 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.018+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c00689e0 con 0x7f22c00d2360 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.019+0000 7f22b67fc640 1 -- 172.21.15.12:0/2190035849 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f22bc006790 con 0x7f22c00d2360 2024-04-15T15:22:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.019+0000 7f22b67fc640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22a406c8f0 0x7f22a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.019+0000 7f22b67fc640 1 -- 172.21.15.12:0/2190035849 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f22bc0ae160 con 0x7f22c00d2360 2024-04-15T15:22:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.019+0000 7f22c545c640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22a406c8f0 0x7f22a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.020+0000 7f22c545c640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22a406c8f0 0x7f22a406edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f22c00ce650 tx=0x7f22b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.024+0000 7f22b67fc640 1 -- 172.21.15.12:0/2190035849 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f22bc07bb00 con 0x7f22c00d2360 2024-04-15T15:22:11.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.206+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f22c00d1190 con 0x7f22c00d2360 2024-04-15T15:22:11.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.207+0000 7f22b67fc640 1 -- 172.21.15.12:0/2190035849 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f22bc07fe00 con 0x7f22c00d2360 2024-04-15T15:22:11.208 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:11.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22a406c8f0 msgr2=0x7f22a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22a406c8f0 0x7f22a406edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f22c00ce650 tx=0x7f22b0008040 comp rx=0 tx=0).stop 2024-04-15T15:22:11.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c00d2360 msgr2=0x7f22c01f7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c00d2360 0x7f22c01f7550 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0090b0 tx=0x7f22bc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 shutdown_connections 2024-04-15T15:22:11.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22a406c8f0 0x7f22a406edb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22c015eae0 0x7f22c00ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22c015e700 0x7f22c00b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.208+0000 7f22c7ec1640 1 --2- 172.21.15.12:0/2190035849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22c00d2360 0x7f22c01f7550 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.209+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 >> 172.21.15.12:0/2190035849 conn(0x7f22c008bf60 msgr2=0x7f22c015f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:11.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.209+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 shutdown_connections 2024-04-15T15:22:11.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.209+0000 7f22c7ec1640 1 -- 172.21.15.12:0/2190035849 wait complete. 2024-04-15T15:22:11.221 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.325+0000 7f5198cf2640 1 Processor -- start 2024-04-15T15:22:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.325+0000 7f5198cf2640 1 -- start start 2024-04-15T15:22:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f5198cf2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f51940d2360 0x7f51940d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f5198cf2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51940b47a0 con 0x7f519415eae0 2024-04-15T15:22:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f5198cf2640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51940b4a80 con 0x7f519415e700 2024-04-15T15:22:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f5198cf2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f51940b53b0 con 0x7f51940d2360 2024-04-15T15:22:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f5192d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f519415eae0 0x7f51940cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34754/0 (socket says 172.21.15.12:34754) 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f5192d9d640 1 -- 172.21.15.12:0/3572354803 learned_addr learned my addr 172.21.15.12:0/3572354803 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f5191d9b640 1 --2- 172.21.15.12:0/3572354803 >> v2:172.21.15.55:3300/0 conn(0x7f51940d2360 0x7f51940d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 573275520 0 0) 0x7f51940b47a0 con 0x7f519415eae0 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f517c003520 con 0x7f519415eae0 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2327809912 0 0) 0x7f517c003520 con 0x7f519415eae0 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 >> v2:172.21.15.55:3300/0 conn(0x7f51940d2360 msgr2=0x7f51940d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 --2- 172.21.15.12:0/3572354803 >> v2:172.21.15.55:3300/0 conn(0x7f51940d2360 0x7f51940d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 >> v1:172.21.15.55:6789/0 conn(0x7f519415e700 legacy=0x7f51940b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51940b4d60 con 0x7f519415eae0 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f518c0019f0 con 0x7f519415eae0 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 >> v1:172.21.15.12:6789/0 conn(0x7f519415eae0 legacy=0x7f51940cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.326+0000 7f519159a640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0039b0 0x7f517c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f519159a640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f517c004800 0x7f517c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f519159a640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c0051c0 0x7f517c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51940b53b0 con 0x7f517c0051c0 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f517c00c090 con 0x7f517c004800 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f517c00c290 con 0x7f517c0039b0 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f5192d9d640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c0051c0 0x7f517c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f5191d9b640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f517c004800 0x7f517c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f519259c640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0039b0 0x7f517c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f5192d9d640 1 -- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0039b0 msgr2=0x7f517c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f5192d9d640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0039b0 0x7f517c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f5192d9d640 1 -- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f517c004800 msgr2=0x7f517c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f5192d9d640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f517c004800 0x7f517c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f5192d9d640 1 -- 172.21.15.12:0/3572354803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517c003520 con 0x7f517c0051c0 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f5191d9b640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f517c004800 0x7f517c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.327+0000 7f519259c640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0039b0 0x7f517c003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f5192d9d640 1 -- 172.21.15.12:0/3572354803 reap_dead start 2024-04-15T15:22:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f5192d9d640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c0051c0 0x7f517c00b9c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f518c00a840 tx=0x7f517c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9706750ef5c6e9cb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c00a460 con 0x7f517c0051c0 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518c034650 con 0x7f517c0051c0 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f519159a640 1 -- 172.21.15.12:0/3572354803 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c034a40 con 0x7f517c0051c0 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f5198cf2640 1 -- 172.21.15.12:0/3572354803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c0051c0 msgr2=0x7f517c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f5198cf2640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c0051c0 0x7f517c00b9c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f518c00a840 tx=0x7f517c005830 comp rx=0 tx=0).stop 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f5198cf2640 1 -- 172.21.15.12:0/3572354803 shutdown_connections 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.328+0000 7f5198cf2640 1 --2- 172.21.15.12:0/3572354803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c0051c0 0x7f517c00b9c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.329+0000 7f5198cf2640 1 -- 172.21.15.12:0/3572354803 >> 172.21.15.12:0/3572354803 conn(0x7f519408bf60 msgr2=0x7f51940d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.329+0000 7f5198cf2640 1 -- 172.21.15.12:0/3572354803 shutdown_connections 2024-04-15T15:22:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.329+0000 7f5198cf2640 1 -- 172.21.15.12:0/3572354803 wait complete. 2024-04-15T15:22:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.330+0000 7f5198cf2640 1 Processor -- start 2024-04-15T15:22:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.330+0000 7f5198cf2640 1 -- start start 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5198cf2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0051c0 0x7f51940b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5198cf2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51940b2140 0x7f51941f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5198cf2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51940b9240 0x7f51940ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5198cf2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51940d1d40 con 0x7f51940b2140 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5198cf2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51940b4450 con 0x7f51940b9240 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5198cf2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f51940d1ec0 con 0x7f517c0051c0 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51940b2140 0x7f51941f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51940b2140 0x7f51941f5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53170/0 (socket says 172.21.15.12:53170) 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 -- 172.21.15.12:0/2270313618 learned_addr learned my addr 172.21.15.12:0/2270313618 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5192d9d640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51940b9240 0x7f51940ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f519259c640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0051c0 0x7f51940b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 -- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0051c0 msgr2=0x7f51940b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0051c0 0x7f51940b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 -- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51940b9240 msgr2=0x7f51940ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51940b9240 0x7f51940ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 -- 172.21.15.12:0/2270313618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51941f5fa0 con 0x7f51940b2140 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f519259c640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0051c0 0x7f51940b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5191d9b640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51940b2140 0x7f51941f5a60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5188011540 tx=0x7f5188011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5182ffd640 1 -- 172.21.15.12:0/2270313618 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188015870 con 0x7f51940b2140 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5182ffd640 1 -- 172.21.15.12:0/2270313618 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5188015a10 con 0x7f51940b2140 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51940b2600 con 0x7f51940b2140 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5182ffd640 1 -- 172.21.15.12:0/2270313618 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188015dc0 con 0x7f51940b2140 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.331+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51940b2bc0 con 0x7f51940b2140 2024-04-15T15:22:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.332+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51940689e0 con 0x7f51940b2140 2024-04-15T15:22:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.333+0000 7f5182ffd640 1 -- 172.21.15.12:0/2270313618 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f518800f460 con 0x7f51940b2140 2024-04-15T15:22:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.333+0000 7f5182ffd640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f517406c8f0 0x7f517406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.334+0000 7f5182ffd640 1 -- 172.21.15.12:0/2270313618 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f518808c8f0 con 0x7f51940b2140 2024-04-15T15:22:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.334+0000 7f519259c640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f517406c8f0 0x7f517406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.334+0000 7f519259c640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f517406c8f0 0x7f517406edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5184000c00 tx=0x7f5184009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.338+0000 7f5182ffd640 1 -- 172.21.15.12:0/2270313618 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f51941f5fa0 con 0x7f51940b2140 2024-04-15T15:22:11.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.536+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5194161de0 con 0x7f51940b2140 2024-04-15T15:22:11.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.537+0000 7f5182ffd640 1 -- 172.21.15.12:0/2270313618 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f5188019030 con 0x7f51940b2140 2024-04-15T15:22:11.538 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f517406c8f0 msgr2=0x7f517406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f517406c8f0 0x7f517406edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5184000c00 tx=0x7f5184009040 comp rx=0 tx=0).stop 2024-04-15T15:22:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51940b2140 msgr2=0x7f51941f5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51940b2140 0x7f51941f5a60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5188011540 tx=0x7f5188011a00 comp rx=0 tx=0).stop 2024-04-15T15:22:11.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 shutdown_connections 2024-04-15T15:22:11.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f517406c8f0 0x7f517406edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51940b9240 0x7f51940ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51940b2140 0x7f51941f5a60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 --2- 172.21.15.12:0/2270313618 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f517c0051c0 0x7f51940b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 >> 172.21.15.12:0/2270313618 conn(0x7f519408bf60 msgr2=0x7f519415f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:11.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 shutdown_connections 2024-04-15T15:22:11.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.539+0000 7f5198cf2640 1 -- 172.21.15.12:0/2270313618 wait complete. 2024-04-15T15:22:11.551 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:22:11.551 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:11.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:11.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.711+0000 7fcb1aa14640 1 Processor -- start 2024-04-15T15:22:11.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.711+0000 7fcb1aa14640 1 -- start start 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.711+0000 7fcb1aa14640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcb140d2360 0x7fcb140d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.711+0000 7fcb1aa14640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 0x7fcb140d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.711+0000 7fcb1aa14640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb140b47a0 con 0x7fcb1415e700 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.711+0000 7fcb1aa14640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcb140b53b0 con 0x7fcb140d2360 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.711+0000 7fcb1aa14640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcb140b4a80 con 0x7fcb1415eae0 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.712+0000 7fcb137fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcb140d2360 0x7fcb140d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.712+0000 7fcb137fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcb140d2360 0x7fcb140d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53180/0 (socket says 172.21.15.12:53180) 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.712+0000 7fcb137fe640 1 -- 172.21.15.12:0/1123718961 learned_addr learned my addr 172.21.15.12:0/1123718961 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.712+0000 7fcb18fb1640 1 --2- 172.21.15.12:0/1123718961 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 0x7fcb140d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.712+0000 7fcb137fe640 1 -- 172.21.15.12:0/1123718961 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 msgr2=0x7fcb140d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.712+0000 7fcb137fe640 1 --2- 172.21.15.12:0/1123718961 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 0x7fcb140d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.712+0000 7fcb137fe640 1 -- 172.21.15.12:0/1123718961 >> v1:172.21.15.55:6789/0 conn(0x7fcb1415e700 legacy=0x7fcb140b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.712+0000 7fcb137fe640 1 -- 172.21.15.12:0/1123718961 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb140b4d60 con 0x7fcb140d2360 2024-04-15T15:22:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb137fe640 1 --2- 172.21.15.12:0/1123718961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb140d2360 0x7fcb140d2b60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009870 tx=0x7fcb0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6678b2d5626861ec server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb12ffd640 1 -- 172.21.15.12:0/1123718961 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08034070 con 0x7fcb140d2360 2024-04-15T15:22:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb12ffd640 1 -- 172.21.15.12:0/1123718961 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb08030840 con 0x7fcb140d2360 2024-04-15T15:22:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb12ffd640 1 -- 172.21.15.12:0/1123718961 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08030c10 con 0x7fcb140d2360 2024-04-15T15:22:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb1aa14640 1 -- 172.21.15.12:0/1123718961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb140d2360 msgr2=0x7fcb140d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/1123718961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb140d2360 0x7fcb140d2b60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009870 tx=0x7fcb0802b980 comp rx=0 tx=0).stop 2024-04-15T15:22:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb1aa14640 1 -- 172.21.15.12:0/1123718961 shutdown_connections 2024-04-15T15:22:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/1123718961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb140d2360 0x7fcb140d2b60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/1123718961 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 0x7fcb140d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.713+0000 7fcb1aa14640 1 -- 172.21.15.12:0/1123718961 >> 172.21.15.12:0/1123718961 conn(0x7fcb1408bf60 msgr2=0x7fcb140d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.714+0000 7fcb1aa14640 1 -- 172.21.15.12:0/1123718961 shutdown_connections 2024-04-15T15:22:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.714+0000 7fcb1aa14640 1 -- 172.21.15.12:0/1123718961 wait complete. 2024-04-15T15:22:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.715+0000 7fcb1aa14640 1 Processor -- start 2024-04-15T15:22:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.715+0000 7fcb1aa14640 1 -- start start 2024-04-15T15:22:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb1aa14640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb140d2360 0x7fcb141614d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb1aa14640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb1415e700 0x7fcb14161a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb1aa14640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 0x7fcb1415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb1aa14640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcb140d1d40 con 0x7fcb1415e700 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb1aa14640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcb140b4450 con 0x7fcb140d2360 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb1aa14640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcb140d1ec0 con 0x7fcb1415eae0 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb1415e700 0x7fcb14161a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb1415e700 0x7fcb14161a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53186/0 (socket says 172.21.15.12:53186) 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 -- 172.21.15.12:0/607823130 learned_addr learned my addr 172.21.15.12:0/607823130 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb13fff640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb140d2360 0x7fcb141614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb18fb1640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 0x7fcb1415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 -- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 msgr2=0x7fcb1415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 0x7fcb1415ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 -- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb140d2360 msgr2=0x7fcb141614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb140d2360 0x7fcb141614d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 -- 172.21.15.12:0/607823130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb140cc8f0 con 0x7fcb1415e700 2024-04-15T15:22:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb137fe640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb1415e700 0x7fcb14161a10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcb0802b9c0 tx=0x7fcb08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb10ff9640 1 -- 172.21.15.12:0/607823130 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08048070 con 0x7fcb1415e700 2024-04-15T15:22:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb10ff9640 1 -- 172.21.15.12:0/607823130 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb08030640 con 0x7fcb1415e700 2024-04-15T15:22:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.716+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb140ccbc0 con 0x7fcb1415e700 2024-04-15T15:22:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.717+0000 7fcb10ff9640 1 -- 172.21.15.12:0/607823130 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08034040 con 0x7fcb1415e700 2024-04-15T15:22:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.717+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb140cd180 con 0x7fcb1415e700 2024-04-15T15:22:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.717+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb140d4860 con 0x7fcb1415e700 2024-04-15T15:22:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.718+0000 7fcb10ff9640 1 -- 172.21.15.12:0/607823130 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb0804b070 con 0x7fcb1415e700 2024-04-15T15:22:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.718+0000 7fcb10ff9640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcb0006c8f0 0x7fcb0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.719+0000 7fcb10ff9640 1 -- 172.21.15.12:0/607823130 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fcb080ade20 con 0x7fcb1415e700 2024-04-15T15:22:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.719+0000 7fcb13fff640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcb0006c8f0 0x7fcb0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.719+0000 7fcb13fff640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcb0006c8f0 0x7fcb0006edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb04000fb0 tx=0x7fcb04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:11.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.723+0000 7fcb10ff9640 1 -- 172.21.15.12:0/607823130 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb140d4860 con 0x7fcb1415e700 2024-04-15T15:22:11.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.910+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb140689e0 con 0x7fcb1415e700 2024-04-15T15:22:11.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.911+0000 7fcb10ff9640 1 -- 172.21.15.12:0/607823130 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7fcb0807bea0 con 0x7fcb1415e700 2024-04-15T15:22:11.912 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:11.913 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcb0006c8f0 msgr2=0x7fcb0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcb0006c8f0 0x7fcb0006edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb04000fb0 tx=0x7fcb04009040 comp rx=0 tx=0).stop 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb1415e700 msgr2=0x7fcb14161a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb1415e700 0x7fcb14161a10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcb0802b9c0 tx=0x7fcb08004730 comp rx=0 tx=0).stop 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 shutdown_connections 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcb1415eae0 0x7fcb1415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb1415e700 0x7fcb14161a10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcb0006c8f0 0x7fcb0006edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 --2- 172.21.15.12:0/607823130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb140d2360 0x7fcb141614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 >> 172.21.15.12:0/607823130 conn(0x7fcb1408bf60 msgr2=0x7fcb140d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 shutdown_connections 2024-04-15T15:22:11.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:11.913+0000 7fcb1aa14640 1 -- 172.21.15.12:0/607823130 wait complete. 2024-04-15T15:22:11.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:11.926 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:12.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.037+0000 7fd9e34d3640 1 Processor -- start 2024-04-15T15:22:12.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.038+0000 7fd9e34d3640 1 -- start start 2024-04-15T15:22:12.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.038+0000 7fd9e34d3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd9dc0d2360 0x7fd9dc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.038+0000 7fd9e34d3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9dc15eae0 0x7fd9dc0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.038+0000 7fd9e34d3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9dc0b47a0 con 0x7fd9dc15e700 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.038+0000 7fd9e34d3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd9dc0b53b0 con 0x7fd9dc15eae0 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.038+0000 7fd9e34d3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd9dc0b4a80 con 0x7fd9dc0d2360 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e1a70640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9dc15eae0 0x7fd9dc0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e1a70640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9dc15eae0 0x7fd9dc0d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53198/0 (socket says 172.21.15.12:53198) 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e1a70640 1 -- 172.21.15.12:0/740489298 learned_addr learned my addr 172.21.15.12:0/740489298 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e126f640 1 --1- 172.21.15.12:0/740489298 >> v1:172.21.15.12:6789/0 conn(0x7fd9dc15e700 0x7fd9dc0b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34764/0 (socket says 172.21.15.12:34764) 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e0a6e640 1 --2- 172.21.15.12:0/740489298 >> v2:172.21.15.55:3300/0 conn(0x7fd9dc0d2360 0x7fd9dc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e1a70640 1 -- 172.21.15.12:0/740489298 >> v2:172.21.15.55:3300/0 conn(0x7fd9dc0d2360 msgr2=0x7fd9dc0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e1a70640 1 --2- 172.21.15.12:0/740489298 >> v2:172.21.15.55:3300/0 conn(0x7fd9dc0d2360 0x7fd9dc0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e1a70640 1 -- 172.21.15.12:0/740489298 >> v1:172.21.15.12:6789/0 conn(0x7fd9dc15e700 legacy=0x7fd9dc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e1a70640 1 -- 172.21.15.12:0/740489298 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9dc0b4d60 con 0x7fd9dc15eae0 2024-04-15T15:22:12.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9d3fff640 1 -- 172.21.15.12:0/740489298 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588535152 0 0) 0x7fd9dc0b47a0 con 0x7fd9dc15e700 2024-04-15T15:22:12.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.039+0000 7fd9e0a6e640 1 --2- 172.21.15.12:0/740489298 >> v2:172.21.15.55:3300/0 conn(0x7fd9dc0d2360 0x7fd9dc0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.040+0000 7fd9e1a70640 1 --2- 172.21.15.12:0/740489298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc15eae0 0x7fd9dc0d30a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc08c7b0 tx=0x7fd9d402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced5b4de255746d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.040+0000 7fd9d3fff640 1 -- 172.21.15.12:0/740489298 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d402cb20 con 0x7fd9dc15eae0 2024-04-15T15:22:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.040+0000 7fd9d3fff640 1 -- 172.21.15.12:0/740489298 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d402ccc0 con 0x7fd9dc15eae0 2024-04-15T15:22:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.040+0000 7fd9d3fff640 1 -- 172.21.15.12:0/740489298 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4034610 con 0x7fd9dc15eae0 2024-04-15T15:22:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.041+0000 7fd9e34d3640 1 -- 172.21.15.12:0/740489298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc15eae0 msgr2=0x7fd9dc0d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.041+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/740489298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc15eae0 0x7fd9dc0d30a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc08c7b0 tx=0x7fd9d402bf80 comp rx=0 tx=0).stop 2024-04-15T15:22:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.041+0000 7fd9e34d3640 1 -- 172.21.15.12:0/740489298 shutdown_connections 2024-04-15T15:22:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.041+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/740489298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc15eae0 0x7fd9dc0d30a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.041+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/740489298 >> v2:172.21.15.55:3300/0 conn(0x7fd9dc0d2360 0x7fd9dc0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.041+0000 7fd9e34d3640 1 -- 172.21.15.12:0/740489298 >> 172.21.15.12:0/740489298 conn(0x7fd9dc08bf60 msgr2=0x7fd9dc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.041+0000 7fd9e34d3640 1 -- 172.21.15.12:0/740489298 shutdown_connections 2024-04-15T15:22:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.041+0000 7fd9e34d3640 1 -- 172.21.15.12:0/740489298 wait complete. 2024-04-15T15:22:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.042+0000 7fd9e34d3640 1 Processor -- start 2024-04-15T15:22:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.042+0000 7fd9e34d3640 1 -- start start 2024-04-15T15:22:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e34d3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc0d2360 0x7fd9dc0ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e34d3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9dc15e700 0x7fd9dc0b9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e34d3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9dc15eae0 0x7fd9dc15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e34d3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd9dc0d1d40 con 0x7fd9dc0d2360 2024-04-15T15:22:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e34d3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd9dc0b4450 con 0x7fd9dc15eae0 2024-04-15T15:22:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e34d3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd9dc0d1ec0 con 0x7fd9dc15e700 2024-04-15T15:22:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc0d2360 0x7fd9dc0ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc0d2360 0x7fd9dc0ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53214/0 (socket says 172.21.15.12:53214) 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 -- 172.21.15.12:0/184798123 learned_addr learned my addr 172.21.15.12:0/184798123 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e0a6e640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9dc15e700 0x7fd9dc0b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e1a70640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9dc15eae0 0x7fd9dc15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 -- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9dc15e700 msgr2=0x7fd9dc0b9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9dc15e700 0x7fd9dc0b9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 -- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9dc15eae0 msgr2=0x7fd9dc15cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9dc15eae0 0x7fd9dc15cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 -- 172.21.15.12:0/184798123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9dc0caee0 con 0x7fd9dc0d2360 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e0a6e640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9dc15e700 0x7fd9dc0b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.043+0000 7fd9e126f640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc0d2360 0x7fd9dc0ca9a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800c4d0 tx=0x7fd9d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.044+0000 7fd9d1ffb640 1 -- 172.21.15.12:0/184798123 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8007480 con 0x7fd9dc0d2360 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.044+0000 7fd9d1ffb640 1 -- 172.21.15.12:0/184798123 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d80057a0 con 0x7fd9dc0d2360 2024-04-15T15:22:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.044+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9dc0b9fd0 con 0x7fd9dc0d2360 2024-04-15T15:22:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.044+0000 7fd9d1ffb640 1 -- 172.21.15.12:0/184798123 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8007620 con 0x7fd9dc0d2360 2024-04-15T15:22:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.044+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9dc0cbcb0 con 0x7fd9dc0d2360 2024-04-15T15:22:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.045+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9dc0689e0 con 0x7fd9dc0d2360 2024-04-15T15:22:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.045+0000 7fd9d1ffb640 1 -- 172.21.15.12:0/184798123 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9d8005b50 con 0x7fd9dc0d2360 2024-04-15T15:22:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.046+0000 7fd9d1ffb640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd9c006c8f0 0x7fd9c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.046+0000 7fd9d1ffb640 1 -- 172.21.15.12:0/184798123 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fd9d808bc50 con 0x7fd9dc0d2360 2024-04-15T15:22:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.046+0000 7fd9e0a6e640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd9c006c8f0 0x7fd9c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.047+0000 7fd9e0a6e640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd9c006c8f0 0x7fd9c006edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0ce650 tx=0x7fd9cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.050+0000 7fd9d1ffb640 1 -- 172.21.15.12:0/184798123 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9d8091020 con 0x7fd9dc0d2360 2024-04-15T15:22:12.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.237+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd9dc0ce9b0 con 0x7fd9dc0d2360 2024-04-15T15:22:12.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.237+0000 7fd9d1ffb640 1 -- 172.21.15.12:0/184798123 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fd9dc0caee0 con 0x7fd9dc0d2360 2024-04-15T15:22:12.239 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd9c006c8f0 msgr2=0x7fd9c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd9c006c8f0 0x7fd9c006edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0ce650 tx=0x7fd9cc008040 comp rx=0 tx=0).stop 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc0d2360 msgr2=0x7fd9dc0ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc0d2360 0x7fd9dc0ca9a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800c4d0 tx=0x7fd9d800c990 comp rx=0 tx=0).stop 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 shutdown_connections 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd9c006c8f0 0x7fd9c006edb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9dc15eae0 0x7fd9dc15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9dc15e700 0x7fd9dc0b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 --2- 172.21.15.12:0/184798123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9dc0d2360 0x7fd9dc0ca9a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 >> 172.21.15.12:0/184798123 conn(0x7fd9dc08bf60 msgr2=0x7fd9dc0b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:12.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 shutdown_connections 2024-04-15T15:22:12.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.239+0000 7fd9e34d3640 1 -- 172.21.15.12:0/184798123 wait complete. 2024-04-15T15:22:12.252 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:12.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.357+0000 7f38db574640 1 Processor -- start 2024-04-15T15:22:12.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.358+0000 7f38db574640 1 -- start start 2024-04-15T15:22:12.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.358+0000 7f38db574640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f38d40d2360 0x7f38d40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.358+0000 7f38db574640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38d40b47a0 con 0x7f38d415e700 2024-04-15T15:22:12.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.358+0000 7f38db574640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38d40b4a80 con 0x7f38d415eae0 2024-04-15T15:22:12.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.358+0000 7f38db574640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f38d40b53b0 con 0x7f38d40d2360 2024-04-15T15:22:12.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.359+0000 7f38d8b0f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f38d40d2360 0x7f38d40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.359+0000 7f38d9310640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f38d415e700 0x7f38d40b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50828/0 (socket says 172.21.15.12:50828) 2024-04-15T15:22:12.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.359+0000 7f38d9310640 1 -- 172.21.15.12:0/1099106294 learned_addr learned my addr 172.21.15.12:0/1099106294 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:12.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.359+0000 7f38d8b0f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f38d40d2360 0x7f38d40d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55474/0 (socket says 172.21.15.12:55474) 2024-04-15T15:22:12.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.359+0000 7f38d8b0f640 1 -- 172.21.15.12:0/1099106294 >> v1:172.21.15.55:6789/0 conn(0x7f38d415eae0 legacy=0x7f38d40cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:12.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.359+0000 7f38d8b0f640 1 -- 172.21.15.12:0/1099106294 >> v1:172.21.15.12:6789/0 conn(0x7f38d415e700 legacy=0x7f38d40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.359+0000 7f38d8b0f640 1 -- 172.21.15.12:0/1099106294 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d40b4d60 con 0x7f38d40d2360 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38d8b0f640 1 --2- 172.21.15.12:0/1099106294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d40d2360 0x7f38d40d2b60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f38d40b5530 tx=0x7f38c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed21aa36c5dbf07 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38cbfff640 1 -- 172.21.15.12:0/1099106294 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4035020 con 0x7f38d40d2360 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38cbfff640 1 -- 172.21.15.12:0/1099106294 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c4031830 con 0x7f38d40d2360 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38cbfff640 1 -- 172.21.15.12:0/1099106294 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4031c00 con 0x7f38d40d2360 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38db574640 1 -- 172.21.15.12:0/1099106294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d40d2360 msgr2=0x7f38d40d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38db574640 1 --2- 172.21.15.12:0/1099106294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d40d2360 0x7f38d40d2b60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f38d40b5530 tx=0x7f38c402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38db574640 1 -- 172.21.15.12:0/1099106294 shutdown_connections 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38db574640 1 --2- 172.21.15.12:0/1099106294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d40d2360 0x7f38d40d2b60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38db574640 1 -- 172.21.15.12:0/1099106294 >> 172.21.15.12:0/1099106294 conn(0x7f38d408bf60 msgr2=0x7f38d40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:12.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38db574640 1 -- 172.21.15.12:0/1099106294 shutdown_connections 2024-04-15T15:22:12.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.360+0000 7f38db574640 1 -- 172.21.15.12:0/1099106294 wait complete. 2024-04-15T15:22:12.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38db574640 1 Processor -- start 2024-04-15T15:22:12.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38db574640 1 -- start start 2024-04-15T15:22:12.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38db574640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38d40d2360 0x7f38d40c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38db574640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d415e700 0x7f38d41614d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38db574640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f38d415eae0 0x7f38d41f5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38db574640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f38d40d1d40 con 0x7f38d415e700 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38db574640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f38d40b4450 con 0x7f38d40d2360 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38db574640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f38d40d1ec0 con 0x7f38d415eae0 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38d8b0f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d415e700 0x7f38d41614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38d8b0f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d415e700 0x7f38d41614d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55482/0 (socket says 172.21.15.12:55482) 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.362+0000 7f38d8b0f640 1 -- 172.21.15.12:0/247436642 learned_addr learned my addr 172.21.15.12:0/247436642 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38d9b11640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f38d415eae0 0x7f38d41f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38d9310640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38d40d2360 0x7f38d40c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38d8b0f640 1 -- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f38d415eae0 msgr2=0x7f38d41f5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38d8b0f640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f38d415eae0 0x7f38d41f5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38d8b0f640 1 -- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38d40d2360 msgr2=0x7f38d40c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38d8b0f640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38d40d2360 0x7f38d40c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38d8b0f640 1 -- 172.21.15.12:0/247436642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d40c82a0 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38d8b0f640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d415e700 0x7f38d41614d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f38c402bee0 tx=0x7f38c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38c9ffb640 1 -- 172.21.15.12:0/247436642 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4041070 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38c9ffb640 1 -- 172.21.15.12:0/247436642 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c403a950 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d41f5950 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38c9ffb640 1 -- 172.21.15.12:0/247436642 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4035040 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.363+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d40ca370 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.364+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d40689e0 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.365+0000 7f38c9ffb640 1 -- 172.21.15.12:0/247436642 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f38c4038020 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.365+0000 7f38c9ffb640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f38b806c8f0 0x7f38b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.365+0000 7f38c9ffb640 1 -- 172.21.15.12:0/247436642 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f38c40adad0 con 0x7f38d415e700 2024-04-15T15:22:12.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.365+0000 7f38d9310640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f38b806c8f0 0x7f38b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.366+0000 7f38d9310640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f38b806c8f0 0x7f38b806edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f38d415b5a0 tx=0x7f38d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.370+0000 7f38c9ffb640 1 -- 172.21.15.12:0/247436642 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38d40c82a0 con 0x7f38d415e700 2024-04-15T15:22:12.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.584+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f38d40ba500 con 0x7f38d415e700 2024-04-15T15:22:12.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.584+0000 7f38c9ffb640 1 -- 172.21.15.12:0/247436642 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f38c407bac0 con 0x7f38d415e700 2024-04-15T15:22:12.586 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.586+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f38b806c8f0 msgr2=0x7f38b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.586+0000 7f38db574640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f38b806c8f0 0x7f38b806edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f38d415b5a0 tx=0x7f38d0009040 comp rx=0 tx=0).stop 2024-04-15T15:22:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.586+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d415e700 msgr2=0x7f38d41614d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.586+0000 7f38db574640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d415e700 0x7f38d41614d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f38c402bee0 tx=0x7f38c40046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.587+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 shutdown_connections 2024-04-15T15:22:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.587+0000 7f38db574640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f38d415eae0 0x7f38d41f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.587+0000 7f38db574640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f38d415e700 0x7f38d41614d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.587+0000 7f38db574640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f38b806c8f0 0x7f38b806edb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.587+0000 7f38db574640 1 --2- 172.21.15.12:0/247436642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38d40d2360 0x7f38d40c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.587+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 >> 172.21.15.12:0/247436642 conn(0x7f38d408bf60 msgr2=0x7f38d40b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:12.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.587+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 shutdown_connections 2024-04-15T15:22:12.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.587+0000 7f38db574640 1 -- 172.21.15.12:0/247436642 wait complete. 2024-04-15T15:22:12.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.706+0000 7fd368311640 1 Processor -- start 2024-04-15T15:22:12.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.706+0000 7fd368311640 1 -- start start 2024-04-15T15:22:12.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.706+0000 7fd368311640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd3600d3240 0x7fd3600d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd368311640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3600d2480 con 0x7fd36015e790 2024-04-15T15:22:12.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd368311640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3600d2760 con 0x7fd3600d1440 2024-04-15T15:22:12.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd368311640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd3600d2db0 con 0x7fd3600d3240 2024-04-15T15:22:12.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3660ad640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd36015e790 0x7fd3600cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50842/0 (socket says 172.21.15.12:50842) 2024-04-15T15:22:12.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3660ad640 1 -- 172.21.15.12:0/1511110838 learned_addr learned my addr 172.21.15.12:0/1511110838 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:12.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3658ac640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd3600d3240 0x7fd3600d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3665984696 0 0) 0x7fd3600d2480 con 0x7fd36015e790 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd34c003520 con 0x7fd36015e790 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3263104079 0 0) 0x7fd34c003520 con 0x7fd36015e790 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd3600d3240 msgr2=0x7fd3600d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd3600d3240 0x7fd3600d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 >> v1:172.21.15.55:6789/0 conn(0x7fd3600d1440 legacy=0x7fd3600b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3600d2a40 con 0x7fd36015e790 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fd35c001670 con 0x7fd36015e790 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 >> v1:172.21.15.12:6789/0 conn(0x7fd36015e790 legacy=0x7fd3600cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd34c0039e0 0x7fd34c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd34c004800 0x7fd34c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd3600d3240 0x7fd34c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd3600d2db0 con 0x7fd34c0039e0 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c00ba70 con 0x7fd34c004800 2024-04-15T15:22:12.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd34c00bc70 con 0x7fd3600d3240 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.707+0000 7fd3660ad640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd34c0039e0 0x7fd34c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.708+0000 7fd3658ac640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd34c004800 0x7fd34c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.708+0000 7fd3668ae640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd3600d3240 0x7fd34c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.708+0000 7fd3660ad640 1 -- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd3600d3240 msgr2=0x7fd34c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.708+0000 7fd3660ad640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd3600d3240 0x7fd34c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.708+0000 7fd3660ad640 1 -- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd34c004800 msgr2=0x7fd34c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.708+0000 7fd3660ad640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd34c004800 0x7fd34c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.708+0000 7fd3660ad640 1 -- 172.21.15.12:0/1511110838 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c003520 con 0x7fd34c0039e0 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd3660ad640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd34c0039e0 0x7fd34c003e20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00c210 tx=0x7fd34c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83753abcfd2a1fe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c007cf0 con 0x7fd34c0039e0 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd35c030070 con 0x7fd34c0039e0 2024-04-15T15:22:12.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd3650ab640 1 -- 172.21.15.12:0/1511110838 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c005d10 con 0x7fd34c0039e0 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd368311640 1 -- 172.21.15.12:0/1511110838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd34c0039e0 msgr2=0x7fd34c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd368311640 1 --2- 172.21.15.12:0/1511110838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd34c0039e0 0x7fd34c003e20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00c210 tx=0x7fd34c005830 comp rx=0 tx=0).stop 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd3668ae640 1 -- 172.21.15.12:0/1511110838 reap_dead start 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd368311640 1 -- 172.21.15.12:0/1511110838 shutdown_connections 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd368311640 1 -- 172.21.15.12:0/1511110838 >> 172.21.15.12:0/1511110838 conn(0x7fd36008bf60 msgr2=0x7fd3600d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd368311640 1 -- 172.21.15.12:0/1511110838 shutdown_connections 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.709+0000 7fd368311640 1 -- 172.21.15.12:0/1511110838 wait complete. 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.710+0000 7fd368311640 1 Processor -- start 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd368311640 1 -- start start 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd368311640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd34c0039e0 0x7fd36015fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd368311640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3600cac00 0x7fd3600a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd368311640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3601614c0 0x7fd3600cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd368311640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd3600d5180 con 0x7fd3601614c0 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd368311640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd3600d0430 con 0x7fd3600cac00 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd368311640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd3600d5300 con 0x7fd34c0039e0 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd3668ae640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3601614c0 0x7fd3600cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd3668ae640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3601614c0 0x7fd3600cc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55488/0 (socket says 172.21.15.12:55488) 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.711+0000 7fd3668ae640 1 -- 172.21.15.12:0/2760196877 learned_addr learned my addr 172.21.15.12:0/2760196877 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3658ac640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3600cac00 0x7fd3600a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3660ad640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd34c0039e0 0x7fd36015fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3668ae640 1 -- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd34c0039e0 msgr2=0x7fd36015fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3668ae640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd34c0039e0 0x7fd36015fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3668ae640 1 -- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3600cac00 msgr2=0x7fd3600a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3668ae640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3600cac00 0x7fd3600a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3668ae640 1 -- 172.21.15.12:0/2760196877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3600a1fa0 con 0x7fd3601614c0 2024-04-15T15:22:12.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3658ac640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3600cac00 0x7fd3600a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd3668ae640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3601614c0 0x7fd3600cc7c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd3600d2220 tx=0x7fd35800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd356ffd640 1 -- 172.21.15.12:0/2760196877 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35801a070 con 0x7fd3601614c0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd356ffd640 1 -- 172.21.15.12:0/2760196877 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd358006d60 con 0x7fd3601614c0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3600c9b60 con 0x7fd3601614c0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd356ffd640 1 -- 172.21.15.12:0/2760196877 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358003b90 con 0x7fd3601614c0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.712+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3600ca120 con 0x7fd3601614c0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.713+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3600b9230 con 0x7fd3601614c0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.713+0000 7fd356ffd640 1 -- 172.21.15.12:0/2760196877 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd358005070 con 0x7fd3601614c0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.714+0000 7fd356ffd640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd34406c920 0x7fd34406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.714+0000 7fd356ffd640 1 -- 172.21.15.12:0/2760196877 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fd35808b480 con 0x7fd3601614c0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.714+0000 7fd3660ad640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd34406c920 0x7fd34406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:12.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.715+0000 7fd3660ad640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd34406c920 0x7fd34406ede0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd35c034000 tx=0x7fd35c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:12.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.718+0000 7fd356ffd640 1 -- 172.21.15.12:0/2760196877 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd3600b9230 con 0x7fd3601614c0 2024-04-15T15:22:12.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.903+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd3600cafe0 con 0x7fd3601614c0 2024-04-15T15:22:12.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.903+0000 7fd356ffd640 1 -- 172.21.15.12:0/2760196877 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fd358059410 con 0x7fd3601614c0 2024-04-15T15:22:12.905 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:12.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd34406c920 msgr2=0x7fd34406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd34406c920 0x7fd34406ede0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd35c034000 tx=0x7fd35c007000 comp rx=0 tx=0).stop 2024-04-15T15:22:12.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3601614c0 msgr2=0x7fd3600cc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:12.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3601614c0 0x7fd3600cc7c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd3600d2220 tx=0x7fd35800fd40 comp rx=0 tx=0).stop 2024-04-15T15:22:12.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 shutdown_connections 2024-04-15T15:22:12.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3601614c0 0x7fd3600cc7c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd34406c920 0x7fd34406ede0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3600cac00 0x7fd3600a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 --2- 172.21.15.12:0/2760196877 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd34c0039e0 0x7fd36015fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:12.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.905+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 >> 172.21.15.12:0/2760196877 conn(0x7fd36008bf60 msgr2=0x7fd3600d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:12.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.906+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 shutdown_connections 2024-04-15T15:22:12.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:12.906+0000 7fd368311640 1 -- 172.21.15.12:0/2760196877 wait complete. 2024-04-15T15:22:12.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:13.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.026+0000 7f8b6c1e8640 1 Processor -- start 2024-04-15T15:22:13.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.026+0000 7f8b6c1e8640 1 -- start start 2024-04-15T15:22:13.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b6c1e8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b640d3240 0x7f8b640d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b6c1e8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8b640d1440 0x7f8b640b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b6c1e8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b640d2480 con 0x7f8b6415e790 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b6c1e8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8b640d2db0 con 0x7f8b640d3240 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b6c1e8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8b640d2760 con 0x7f8b640d1440 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b69783640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b640d3240 0x7f8b640d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b69783640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b640d3240 0x7f8b640d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55502/0 (socket says 172.21.15.12:55502) 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b69783640 1 -- 172.21.15.12:0/1426683908 learned_addr learned my addr 172.21.15.12:0/1426683908 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b69f84640 1 --1- 172.21.15.12:0/1426683908 >> v1:172.21.15.12:6789/0 conn(0x7f8b6415e790 0x7f8b640cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50848/0 (socket says 172.21.15.12:50848) 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b6a785640 1 --2- 172.21.15.12:0/1426683908 >> v2:172.21.15.55:3300/0 conn(0x7f8b640d1440 0x7f8b640b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b69783640 1 -- 172.21.15.12:0/1426683908 >> v2:172.21.15.55:3300/0 conn(0x7f8b640d1440 msgr2=0x7f8b640b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b69783640 1 --2- 172.21.15.12:0/1426683908 >> v2:172.21.15.55:3300/0 conn(0x7f8b640d1440 0x7f8b640b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b69783640 1 -- 172.21.15.12:0/1426683908 >> v1:172.21.15.12:6789/0 conn(0x7f8b6415e790 legacy=0x7f8b640cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b69783640 1 -- 172.21.15.12:0/1426683908 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b640d2a40 con 0x7f8b640d3240 2024-04-15T15:22:13.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.027+0000 7f8b68f82640 1 -- 172.21.15.12:0/1426683908 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662615239 0 0) 0x7f8b640d2480 con 0x7f8b6415e790 2024-04-15T15:22:13.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.028+0000 7f8b69783640 1 --2- 172.21.15.12:0/1426683908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640d0f00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8b640c8120 tx=0x7f8b5802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a950ecdfda9d91f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.028+0000 7f8b68f82640 1 -- 172.21.15.12:0/1426683908 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5802cb20 con 0x7f8b640d3240 2024-04-15T15:22:13.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.028+0000 7f8b68f82640 1 -- 172.21.15.12:0/1426683908 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b5802ccc0 con 0x7f8b640d3240 2024-04-15T15:22:13.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.028+0000 7f8b68f82640 1 -- 172.21.15.12:0/1426683908 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58034610 con 0x7f8b640d3240 2024-04-15T15:22:13.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.028+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1426683908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 msgr2=0x7f8b640d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.028+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1426683908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640d0f00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8b640c8120 tx=0x7f8b5802bf80 comp rx=0 tx=0).stop 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.029+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1426683908 shutdown_connections 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.029+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1426683908 >> v2:172.21.15.55:3300/0 conn(0x7f8b640d1440 0x7f8b640b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.029+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1426683908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640d0f00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.029+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1426683908 >> 172.21.15.12:0/1426683908 conn(0x7f8b6408bf60 msgr2=0x7f8b640d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.029+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1426683908 shutdown_connections 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.029+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1426683908 wait complete. 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.030+0000 7f8b6c1e8640 1 Processor -- start 2024-04-15T15:22:13.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.030+0000 7f8b6c1e8640 1 -- start start 2024-04-15T15:22:13.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6c1e8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8b640d1440 0x7f8b641f7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6c1e8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6c1e8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b6415e790 0x7f8b640ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6c1e8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8b640d5180 con 0x7f8b640d3240 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6c1e8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8b640d0430 con 0x7f8b6415e790 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6c1e8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8b640d5300 con 0x7f8b640d1440 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640c9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55516/0 (socket says 172.21.15.12:55516) 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 -- 172.21.15.12:0/1861473343 learned_addr learned my addr 172.21.15.12:0/1861473343 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6a785640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b6415e790 0x7f8b640ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69f84640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8b640d1440 0x7f8b641f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 -- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8b640d1440 msgr2=0x7f8b641f7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8b640d1440 0x7f8b641f7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 -- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b6415e790 msgr2=0x7f8b640ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b6415e790 0x7f8b640ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 -- 172.21.15.12:0/1861473343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b640c97f0 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6a785640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b6415e790 0x7f8b640ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b69783640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640c9f10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009f40 tx=0x7f8b58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b627fc640 1 -- 172.21.15.12:0/1861473343 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58043020 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b627fc640 1 -- 172.21.15.12:0/1861473343 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b5803e450 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b64159630 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.031+0000 7f8b627fc640 1 -- 172.21.15.12:0/1861473343 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58009510 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.032+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b640c9ad0 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.033+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b640689e0 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.033+0000 7f8b627fc640 1 -- 172.21.15.12:0/1861473343 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b58046020 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.033+0000 7f8b627fc640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8b5006c8f0 0x7f8b5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.033+0000 7f8b627fc640 1 -- 172.21.15.12:0/1861473343 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f8b580ae7e0 con 0x7f8b640d3240 2024-04-15T15:22:13.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.034+0000 7f8b69f84640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8b5006c8f0 0x7f8b5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.034+0000 7f8b69f84640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8b5006c8f0 0x7f8b5006edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c000f60 tx=0x7f8b5c007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.038+0000 7f8b627fc640 1 -- 172.21.15.12:0/1861473343 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b5804d030 con 0x7f8b640d3240 2024-04-15T15:22:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.226+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8b640d1ce0 con 0x7f8b640d3240 2024-04-15T15:22:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.226+0000 7f8b627fc640 1 -- 172.21.15.12:0/1861473343 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f8b5807c7d0 con 0x7f8b640d3240 2024-04-15T15:22:13.228 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8b5006c8f0 msgr2=0x7f8b5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8b5006c8f0 0x7f8b5006edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c000f60 tx=0x7f8b5c007c00 comp rx=0 tx=0).stop 2024-04-15T15:22:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 msgr2=0x7f8b640c9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640c9f10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009f40 tx=0x7f8b58004a40 comp rx=0 tx=0).stop 2024-04-15T15:22:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 shutdown_connections 2024-04-15T15:22:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8b5006c8f0 0x7f8b5006edb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b6415e790 0x7f8b640ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b640d3240 0x7f8b640c9f10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.228+0000 7f8b6c1e8640 1 --2- 172.21.15.12:0/1861473343 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8b640d1440 0x7f8b641f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.229+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 >> 172.21.15.12:0/1861473343 conn(0x7f8b6408bf60 msgr2=0x7f8b640b9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:13.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.229+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 shutdown_connections 2024-04-15T15:22:13.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.229+0000 7f8b6c1e8640 1 -- 172.21.15.12:0/1861473343 wait complete. 2024-04-15T15:22:13.241 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:13.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.345+0000 7fd77f73d640 1 Processor -- start 2024-04-15T15:22:13.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.346+0000 7fd77f73d640 1 -- start start 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.346+0000 7fd77f73d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd77815e700 0x7fd7780b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.346+0000 7fd77f73d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd7780d3270 0x7fd7780cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.346+0000 7fd77f73d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7780b47a0 con 0x7fd7780d2360 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.346+0000 7fd77f73d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd7780b53b0 con 0x7fd77815e700 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.346+0000 7fd77f73d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd7780b4a80 con 0x7fd7780d3270 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.346+0000 7fd77d4d9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd77815e700 0x7fd7780b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd77ccd8640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd7780d2360 0x7fd7780d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50856/0 (socket says 172.21.15.12:50856) 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd77ccd8640 1 -- 172.21.15.12:0/2019705482 learned_addr learned my addr 172.21.15.12:0/2019705482 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd77dcda640 1 --2- 172.21.15.12:0/2019705482 >> v2:172.21.15.55:3300/0 conn(0x7fd7780d3270 0x7fd7780cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd77d4d9640 1 -- 172.21.15.12:0/2019705482 >> v2:172.21.15.55:3300/0 conn(0x7fd7780d3270 msgr2=0x7fd7780cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd77d4d9640 1 --2- 172.21.15.12:0/2019705482 >> v2:172.21.15.55:3300/0 conn(0x7fd7780d3270 0x7fd7780cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd77d4d9640 1 -- 172.21.15.12:0/2019705482 >> v1:172.21.15.12:6789/0 conn(0x7fd7780d2360 legacy=0x7fd7780d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd76ffff640 1 -- 172.21.15.12:0/2019705482 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765105215 0 0) 0x7fd7780b47a0 con 0x7fd7780d2360 2024-04-15T15:22:13.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd77d4d9640 1 -- 172.21.15.12:0/2019705482 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7780b4d60 con 0x7fd77815e700 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.347+0000 7fd77dcda640 1 --2- 172.21.15.12:0/2019705482 >> v2:172.21.15.55:3300/0 conn(0x7fd7780d3270 0x7fd7780cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77d4d9640 1 --2- 172.21.15.12:0/2019705482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd7780b5840 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd7780d1b00 tx=0x7fd77402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34ac98718c11b1f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd76ffff640 1 -- 172.21.15.12:0/2019705482 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774035020 con 0x7fd77815e700 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd76ffff640 1 -- 172.21.15.12:0/2019705482 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd774031830 con 0x7fd77815e700 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd76ffff640 1 -- 172.21.15.12:0/2019705482 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774031c00 con 0x7fd77815e700 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77f73d640 1 -- 172.21.15.12:0/2019705482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 msgr2=0x7fd7780b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77f73d640 1 --2- 172.21.15.12:0/2019705482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd7780b5840 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd7780d1b00 tx=0x7fd77402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77f73d640 1 -- 172.21.15.12:0/2019705482 shutdown_connections 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77f73d640 1 --2- 172.21.15.12:0/2019705482 >> v2:172.21.15.55:3300/0 conn(0x7fd7780d3270 0x7fd7780cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77f73d640 1 --2- 172.21.15.12:0/2019705482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd7780b5840 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77f73d640 1 -- 172.21.15.12:0/2019705482 >> 172.21.15.12:0/2019705482 conn(0x7fd77808bf60 msgr2=0x7fd7780d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77f73d640 1 -- 172.21.15.12:0/2019705482 shutdown_connections 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.348+0000 7fd77f73d640 1 -- 172.21.15.12:0/2019705482 wait complete. 2024-04-15T15:22:13.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.349+0000 7fd77f73d640 1 Processor -- start 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.349+0000 7fd77f73d640 1 -- start start 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77f73d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7780d2360 0x7fd7780b87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77f73d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7780d3270 0x7fd7780b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77f73d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd778158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77f73d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd7780b5d80 con 0x7fd77815e700 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77f73d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd7780b4450 con 0x7fd7780d3270 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77f73d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd7780d1d40 con 0x7fd7780d2360 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77dcda640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd778158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77dcda640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd778158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55526/0 (socket says 172.21.15.12:55526) 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77dcda640 1 -- 172.21.15.12:0/4197201479 learned_addr learned my addr 172.21.15.12:0/4197201479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.350+0000 7fd77ccd8640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7780d3270 0x7fd7780b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77d4d9640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7780d2360 0x7fd7780b87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77dcda640 1 -- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7780d2360 msgr2=0x7fd7780b87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77dcda640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7780d2360 0x7fd7780b87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77dcda640 1 -- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7780d3270 msgr2=0x7fd7780b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77dcda640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7780d3270 0x7fd7780b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77dcda640 1 -- 172.21.15.12:0/4197201479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd778158fa0 con 0x7fd77815e700 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77dcda640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd778158170 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd77000cf20 tx=0x7fd77000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd76dffb640 1 -- 172.21.15.12:0/4197201479 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77001a070 con 0x7fd77815e700 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd76dffb640 1 -- 172.21.15.12:0/4197201479 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd77000d840 con 0x7fd77815e700 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd76dffb640 1 -- 172.21.15.12:0/4197201479 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77000dc30 con 0x7fd77815e700 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7780b7cc0 con 0x7fd77815e700 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.351+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7780b8280 con 0x7fd77815e700 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.352+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7780b8480 con 0x7fd77815e700 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.353+0000 7fd76dffb640 1 -- 172.21.15.12:0/4197201479 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd77001d070 con 0x7fd77815e700 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.353+0000 7fd76dffb640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd75c06c8f0 0x7fd75c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.353+0000 7fd76dffb640 1 -- 172.21.15.12:0/4197201479 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fd77008c850 con 0x7fd77815e700 2024-04-15T15:22:13.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.353+0000 7fd77d4d9640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd75c06c8f0 0x7fd75c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.354+0000 7fd77d4d9640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd75c06c8f0 0x7fd75c06edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd774036000 tx=0x7fd774008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.357+0000 7fd76dffb640 1 -- 172.21.15.12:0/4197201479 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7780b8480 con 0x7fd77815e700 2024-04-15T15:22:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.554+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd7780689e0 con 0x7fd77815e700 2024-04-15T15:22:13.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.554+0000 7fd76dffb640 1 -- 172.21.15.12:0/4197201479 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fd77005a840 con 0x7fd77815e700 2024-04-15T15:22:13.556 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd75c06c8f0 msgr2=0x7fd75c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd75c06c8f0 0x7fd75c06edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd774036000 tx=0x7fd774008030 comp rx=0 tx=0).stop 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 msgr2=0x7fd778158170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd778158170 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd77000cf20 tx=0x7fd77000a7d0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 shutdown_connections 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd77815e700 0x7fd778158170 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd75c06c8f0 0x7fd75c06edb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7780d3270 0x7fd7780b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 --2- 172.21.15.12:0/4197201479 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd7780d2360 0x7fd7780b87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 >> 172.21.15.12:0/4197201479 conn(0x7fd77808bf60 msgr2=0x7fd7780d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:13.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 shutdown_connections 2024-04-15T15:22:13.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.556+0000 7fd77f73d640 1 -- 172.21.15.12:0/4197201479 wait complete. 2024-04-15T15:22:13.568 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-04-15T15:22:13.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:13.569 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-15T15:22:13.596 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:13.596 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:22:13.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:13.597 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:22:13.652 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:22:13.653 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:13.672 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:22:13.672 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:13.731 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.834+0000 7f7b383c8640 1 Processor -- start 2024-04-15T15:22:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.834+0000 7f7b383c8640 1 -- start start 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b383c8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7b300d5150 0x7f7b300d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b383c8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b3015eb70 con 0x7f7b3015e790 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b383c8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b300d3c80 con 0x7f7b300d15e0 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b383c8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7b300d3790 con 0x7f7b300d5150 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b36164640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7b3015e790 0x7f7b300b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50864/0 (socket says 172.21.15.12:50864) 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b36164640 1 -- 172.21.15.12:0/1118153798 learned_addr learned my addr 172.21.15.12:0/1118153798 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b36965640 1 --2- 172.21.15.12:0/1118153798 >> v2:172.21.15.12:3300/0 conn(0x7f7b300d5150 0x7f7b300d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b35162640 1 -- 172.21.15.12:0/1118153798 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1550457045 0 0) 0x7f7b3015eb70 con 0x7f7b3015e790 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b35162640 1 -- 172.21.15.12:0/1118153798 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b1c003520 con 0x7f7b3015e790 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b36965640 1 -- 172.21.15.12:0/1118153798 >> v1:172.21.15.55:6789/0 conn(0x7f7b300d15e0 legacy=0x7f7b300d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:13.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b36965640 1 -- 172.21.15.12:0/1118153798 >> v1:172.21.15.12:6789/0 conn(0x7f7b3015e790 legacy=0x7f7b300b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.835+0000 7f7b36965640 1 -- 172.21.15.12:0/1118153798 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b300d3f60 con 0x7f7b300d5150 2024-04-15T15:22:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.836+0000 7f7b36965640 1 --2- 172.21.15.12:0/1118153798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d5150 0x7f7b300d1a90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7b300b61c0 tx=0x7f7b2802bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92b399324f7ea44 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.836+0000 7f7b35162640 1 -- 172.21.15.12:0/1118153798 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b28039070 con 0x7f7b300d5150 2024-04-15T15:22:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.836+0000 7f7b35162640 1 -- 172.21.15.12:0/1118153798 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b2802cb20 con 0x7f7b300d5150 2024-04-15T15:22:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.836+0000 7f7b35162640 1 -- 172.21.15.12:0/1118153798 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b28034440 con 0x7f7b300d5150 2024-04-15T15:22:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.837+0000 7f7b383c8640 1 -- 172.21.15.12:0/1118153798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d5150 msgr2=0x7f7b300d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.837+0000 7f7b383c8640 1 --2- 172.21.15.12:0/1118153798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d5150 0x7f7b300d1a90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7b300b61c0 tx=0x7f7b2802bd70 comp rx=0 tx=0).stop 2024-04-15T15:22:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.837+0000 7f7b383c8640 1 -- 172.21.15.12:0/1118153798 shutdown_connections 2024-04-15T15:22:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.837+0000 7f7b383c8640 1 --2- 172.21.15.12:0/1118153798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d5150 0x7f7b300d1a90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.837+0000 7f7b383c8640 1 -- 172.21.15.12:0/1118153798 >> 172.21.15.12:0/1118153798 conn(0x7f7b30067b40 msgr2=0x7f7b30067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.837+0000 7f7b383c8640 1 -- 172.21.15.12:0/1118153798 shutdown_connections 2024-04-15T15:22:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.837+0000 7f7b383c8640 1 -- 172.21.15.12:0/1118153798 wait complete. 2024-04-15T15:22:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.838+0000 7f7b383c8640 1 Processor -- start 2024-04-15T15:22:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.838+0000 7f7b383c8640 1 -- start start 2024-04-15T15:22:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b383c8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d15e0 0x7f7b3015f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b383c8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b300d5150 0x7f7b300ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b383c8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7b3015e790 0x7f7b300ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b383c8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7b300b7db0 con 0x7f7b300d15e0 2024-04-15T15:22:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b383c8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7b300d45b0 con 0x7f7b300d5150 2024-04-15T15:22:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b383c8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7b300d29c0 con 0x7f7b3015e790 2024-04-15T15:22:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d15e0 0x7f7b3015f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d15e0 0x7f7b3015f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55536/0 (socket says 172.21.15.12:55536) 2024-04-15T15:22:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 -- 172.21.15.12:0/2197656359 learned_addr learned my addr 172.21.15.12:0/2197656359 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b35963640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b300d5150 0x7f7b300ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36965640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7b3015e790 0x7f7b300ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 -- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7b3015e790 msgr2=0x7f7b300ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7b3015e790 0x7f7b300ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 -- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b300d5150 msgr2=0x7f7b300ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b300d5150 0x7f7b300ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 -- 172.21.15.12:0/2197656359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3015fac0 con 0x7f7b300d15e0 2024-04-15T15:22:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b36164640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d15e0 0x7f7b3015f580 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00c390 tx=0x7f7b2c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.839+0000 7f7b26ffd640 1 -- 172.21.15.12:0/2197656359 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c017070 con 0x7f7b300d15e0 2024-04-15T15:22:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.840+0000 7f7b26ffd640 1 -- 172.21.15.12:0/2197656359 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b2c005ce0 con 0x7f7b300d15e0 2024-04-15T15:22:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.840+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3015fd60 con 0x7f7b300d15e0 2024-04-15T15:22:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.840+0000 7f7b26ffd640 1 -- 172.21.15.12:0/2197656359 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c0072f0 con 0x7f7b300d15e0 2024-04-15T15:22:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.840+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b30160320 con 0x7f7b300d15e0 2024-04-15T15:22:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.841+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b300689e0 con 0x7f7b300d15e0 2024-04-15T15:22:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.841+0000 7f7b26ffd640 1 -- 172.21.15.12:0/2197656359 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7b2c01a070 con 0x7f7b300d15e0 2024-04-15T15:22:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.841+0000 7f7b26ffd640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7b1406c8f0 0x7f7b1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.841+0000 7f7b26ffd640 1 -- 172.21.15.12:0/2197656359 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f7b2c08b3b0 con 0x7f7b300d15e0 2024-04-15T15:22:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.842+0000 7f7b35963640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7b1406c8f0 0x7f7b1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.842+0000 7f7b35963640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7b1406c8f0 0x7f7b1406edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b300d2da0 tx=0x7f7b20009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:13.846+0000 7f7b26ffd640 1 -- 172.21.15.12:0/2197656359 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7b2c0593a0 con 0x7f7b300d15e0 2024-04-15T15:22:14.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.033+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b300ba0d0 con 0x7f7b300d15e0 2024-04-15T15:22:14.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.034+0000 7f7b26ffd640 1 -- 172.21.15.12:0/2197656359 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f7b2c058ca0 con 0x7f7b300d15e0 2024-04-15T15:22:14.036 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:14.036 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:14.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7b1406c8f0 msgr2=0x7f7b1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7b1406c8f0 0x7f7b1406edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b300d2da0 tx=0x7f7b20009040 comp rx=0 tx=0).stop 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d15e0 msgr2=0x7f7b3015f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d15e0 0x7f7b3015f580 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00c390 tx=0x7f7b2c00c850 comp rx=0 tx=0).stop 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 shutdown_connections 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7b3015e790 0x7f7b300ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7b1406c8f0 0x7f7b1406edb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b300d5150 0x7f7b300ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 --2- 172.21.15.12:0/2197656359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b300d15e0 0x7f7b3015f580 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 >> 172.21.15.12:0/2197656359 conn(0x7f7b30067b40 msgr2=0x7f7b300b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 shutdown_connections 2024-04-15T15:22:14.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.037+0000 7f7b383c8640 1 -- 172.21.15.12:0/2197656359 wait complete. 2024-04-15T15:22:14.050 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:14.050 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.157+0000 7f3ce6946640 1 Processor -- start 2024-04-15T15:22:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.157+0000 7f3ce6946640 1 -- start start 2024-04-15T15:22:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.157+0000 7f3ce6946640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3ce015e700 0x7f3ce00b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.157+0000 7f3ce6946640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3ce00d3270 0x7f3ce00cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.157+0000 7f3ce6946640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce00b47a0 con 0x7f3ce00d2360 2024-04-15T15:22:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.157+0000 7f3ce6946640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce00b53b0 con 0x7f3ce015e700 2024-04-15T15:22:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.157+0000 7f3ce6946640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce00b4a80 con 0x7f3ce00d3270 2024-04-15T15:22:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3ce015e700 0x7f3ce00b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdf7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3ce00d2360 0x7f3ce00d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50868/0 (socket says 172.21.15.12:50868) 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3ce015e700 0x7f3ce00b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55546/0 (socket says 172.21.15.12:55546) 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdf7fe640 1 -- 172.21.15.12:0/3122276761 learned_addr learned my addr 172.21.15.12:0/3122276761 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3ce4ee3640 1 --2- 172.21.15.12:0/3122276761 >> v2:172.21.15.55:3300/0 conn(0x7f3ce00d3270 0x7f3ce00cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdffff640 1 -- 172.21.15.12:0/3122276761 >> v2:172.21.15.55:3300/0 conn(0x7f3ce00d3270 msgr2=0x7f3ce00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdffff640 1 --2- 172.21.15.12:0/3122276761 >> v2:172.21.15.55:3300/0 conn(0x7f3ce00d3270 0x7f3ce00cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdffff640 1 -- 172.21.15.12:0/3122276761 >> v1:172.21.15.12:6789/0 conn(0x7f3ce00d2360 legacy=0x7f3ce00d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdffff640 1 -- 172.21.15.12:0/3122276761 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00b4d60 con 0x7f3ce015e700 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.158+0000 7f3cdeffd640 1 -- 172.21.15.12:0/3122276761 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279919355 0 0) 0x7f3ce00b47a0 con 0x7f3ce00d2360 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3cdffff640 1 --2- 172.21.15.12:0/3122276761 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce015e700 0x7f3ce00b5840 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00d1b00 tx=0x7f3cd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd573ecce05ee442 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3cdeffd640 1 -- 172.21.15.12:0/3122276761 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0035020 con 0x7f3ce015e700 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3cdeffd640 1 -- 172.21.15.12:0/3122276761 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd0031830 con 0x7f3ce015e700 2024-04-15T15:22:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3cdeffd640 1 -- 172.21.15.12:0/3122276761 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0031c00 con 0x7f3ce015e700 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3ce6946640 1 -- 172.21.15.12:0/3122276761 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce015e700 msgr2=0x7f3ce00b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3122276761 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce015e700 0x7f3ce00b5840 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00d1b00 tx=0x7f3cd002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3ce6946640 1 -- 172.21.15.12:0/3122276761 shutdown_connections 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3122276761 >> v2:172.21.15.55:3300/0 conn(0x7f3ce00d3270 0x7f3ce00cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3122276761 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce015e700 0x7f3ce00b5840 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3ce6946640 1 -- 172.21.15.12:0/3122276761 >> 172.21.15.12:0/3122276761 conn(0x7f3ce008bf60 msgr2=0x7f3ce00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3ce6946640 1 -- 172.21.15.12:0/3122276761 shutdown_connections 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.159+0000 7f3ce6946640 1 -- 172.21.15.12:0/3122276761 wait complete. 2024-04-15T15:22:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.161+0000 7f3ce6946640 1 Processor -- start 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.161+0000 7f3ce6946640 1 -- start start 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.161+0000 7f3ce6946640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ce00d2360 0x7f3ce00b9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.161+0000 7f3ce6946640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce00d3270 0x7f3ce00b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.161+0000 7f3ce6946640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ce015e700 0x7f3ce0158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.161+0000 7f3ce6946640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00b5d80 con 0x7f3ce00d3270 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.161+0000 7f3ce6946640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00b4450 con 0x7f3ce00d2360 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.161+0000 7f3ce6946640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00d1d40 con 0x7f3ce015e700 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce00d3270 0x7f3ce00b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce00d3270 0x7f3ce00b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55552/0 (socket says 172.21.15.12:55552) 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 -- 172.21.15.12:0/3625451693 learned_addr learned my addr 172.21.15.12:0/3625451693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdffff640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ce00d2360 0x7f3ce00b9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3ce4ee3640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ce015e700 0x7f3ce0158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 -- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ce015e700 msgr2=0x7f3ce0158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ce015e700 0x7f3ce0158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 -- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ce00d2360 msgr2=0x7f3ce00b9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ce00d2360 0x7f3ce00b9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 -- 172.21.15.12:0/3625451693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00b82e0 con 0x7f3ce00d3270 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdf7fe640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce00d3270 0x7f3ce00b7b60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c4d0 tx=0x7f3cd400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdcff9640 1 -- 172.21.15.12:0/3625451693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4018070 con 0x7f3ce00d3270 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdcff9640 1 -- 172.21.15.12:0/3625451693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd4007500 con 0x7f3ce00d3270 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce00b85c0 con 0x7f3ce00d3270 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3cdcff9640 1 -- 172.21.15.12:0/3625451693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4004a20 con 0x7f3ce00d3270 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.162+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce00b8b80 con 0x7f3ce00d3270 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.163+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce00689e0 con 0x7f3ce00d3270 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.164+0000 7f3cdcff9640 1 -- 172.21.15.12:0/3625451693 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cd401b070 con 0x7f3ce00d3270 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.164+0000 7f3cdcff9640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.164+0000 7f3cdcff9640 1 -- 172.21.15.12:0/3625451693 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f3cd408b1e0 con 0x7f3ce00d3270 2024-04-15T15:22:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.165+0000 7f3cdffff640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.165+0000 7f3cdffff640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0036000 tx=0x7f3cd0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.169+0000 7f3cdcff9640 1 -- 172.21.15.12:0/3625451693 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cd4058b80 con 0x7f3ce00d3270 2024-04-15T15:22:14.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.355+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3ce00ba500 con 0x7f3ce00d3270 2024-04-15T15:22:14.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.355+0000 7f3cdcff9640 1 -- 172.21.15.12:0/3625451693 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f3cd405ce80 con 0x7f3ce00d3270 2024-04-15T15:22:14.357 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:14.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ccc06c8f0 msgr2=0x7f3ccc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0036000 tx=0x7f3cd0008030 comp rx=0 tx=0).stop 2024-04-15T15:22:14.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce00d3270 msgr2=0x7f3ce00b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce00d3270 0x7f3ce00b7b60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c4d0 tx=0x7f3cd400c990 comp rx=0 tx=0).stop 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 shutdown_connections 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3ce015e700 0x7f3ce0158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ce00d3270 0x7f3ce00b7b60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ccc06c8f0 0x7f3ccc06edb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 --2- 172.21.15.12:0/3625451693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ce00d2360 0x7f3ce00b9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 >> 172.21.15.12:0/3625451693 conn(0x7f3ce008bf60 msgr2=0x7f3ce00cd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 shutdown_connections 2024-04-15T15:22:14.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.358+0000 7f3ce6946640 1 -- 172.21.15.12:0/3625451693 wait complete. 2024-04-15T15:22:14.370 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:14.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.476+0000 7f7eec3a8640 1 Processor -- start 2024-04-15T15:22:14.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.476+0000 7f7eec3a8640 1 -- start start 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.477+0000 7f7eec3a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee415e790 0x7f7ee40b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.477+0000 7f7eec3a8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ee40b7be0 0x7f7ee40d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.477+0000 7f7eec3a8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ee415eb70 con 0x7f7ee40d1b20 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.477+0000 7f7eec3a8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7ee40d3790 con 0x7f7ee40b7be0 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.477+0000 7f7eec3a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee40d45b0 con 0x7f7ee415e790 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.477+0000 7f7ee9943640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ee40b7be0 0x7f7ee40d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.477+0000 7f7eea945640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7ee40d1b20 0x7f7ee40d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50870/0 (socket says 172.21.15.12:50870) 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.477+0000 7f7eea945640 1 -- 172.21.15.12:0/2433801575 learned_addr learned my addr 172.21.15.12:0/2433801575 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.478+0000 7f7eea144640 1 --2- 172.21.15.12:0/2433801575 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee415e790 0x7f7ee40b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.478+0000 7f7ee9943640 1 -- 172.21.15.12:0/2433801575 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee415e790 msgr2=0x7f7ee40b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:14.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.478+0000 7f7ee9943640 1 --2- 172.21.15.12:0/2433801575 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee415e790 0x7f7ee40b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.478+0000 7f7ee9943640 1 -- 172.21.15.12:0/2433801575 >> v1:172.21.15.12:6789/0 conn(0x7f7ee40d1b20 legacy=0x7f7ee40d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.478+0000 7f7ee9943640 1 -- 172.21.15.12:0/2433801575 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee40d3c80 con 0x7f7ee40b7be0 2024-04-15T15:22:14.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.478+0000 7f7ee9142640 1 -- 172.21.15.12:0/2433801575 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2653170510 0 0) 0x7f7ee415eb70 con 0x7f7ee40d1b20 2024-04-15T15:22:14.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.478+0000 7f7eea144640 1 --2- 172.21.15.12:0/2433801575 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee415e790 0x7f7ee40b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:14.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.478+0000 7f7ee9943640 1 --2- 172.21.15.12:0/2433801575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee40b7be0 0x7f7ee40d15e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80097a0 tx=0x7f7ed802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c66add0e0a0e242 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7ee9142640 1 -- 172.21.15.12:0/2433801575 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8038070 con 0x7f7ee40b7be0 2024-04-15T15:22:14.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7ee9142640 1 -- 172.21.15.12:0/2433801575 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed8031700 con 0x7f7ee40b7be0 2024-04-15T15:22:14.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7ee9142640 1 -- 172.21.15.12:0/2433801575 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8031ad0 con 0x7f7ee40b7be0 2024-04-15T15:22:14.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2433801575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee40b7be0 msgr2=0x7f7ee40d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2433801575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee40b7be0 0x7f7ee40d15e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80097a0 tx=0x7f7ed802b7a0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2433801575 shutdown_connections 2024-04-15T15:22:14.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2433801575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee40b7be0 0x7f7ee40d15e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2433801575 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee415e790 0x7f7ee40b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2433801575 >> 172.21.15.12:0/2433801575 conn(0x7f7ee4067b40 msgr2=0x7f7ee4067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:14.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2433801575 shutdown_connections 2024-04-15T15:22:14.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.479+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2433801575 wait complete. 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.480+0000 7f7eec3a8640 1 Processor -- start 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.480+0000 7f7eec3a8640 1 -- start start 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eec3a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee40b7be0 0x7f7ee414cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eec3a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ee40d1b20 0x7f7ee40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eec3a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee415e790 0x7f7ee40ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eec3a8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40d5150 con 0x7f7ee415e790 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eec3a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40d4240 con 0x7f7ee40d1b20 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eec3a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee40d52d0 con 0x7f7ee40b7be0 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eea945640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee415e790 0x7f7ee40ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eea945640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee415e790 0x7f7ee40ca720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55568/0 (socket says 172.21.15.12:55568) 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eea945640 1 -- 172.21.15.12:0/2565091919 learned_addr learned my addr 172.21.15.12:0/2565091919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7ee9943640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ee40d1b20 0x7f7ee40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eea144640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee40b7be0 0x7f7ee414cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eea945640 1 -- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee40b7be0 msgr2=0x7f7ee414cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eea945640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee40b7be0 0x7f7ee414cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eea945640 1 -- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ee40d1b20 msgr2=0x7f7ee40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.481+0000 7f7eea945640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ee40d1b20 0x7f7ee40ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.482+0000 7f7eea945640 1 -- 172.21.15.12:0/2565091919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee414d070 con 0x7f7ee415e790 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.482+0000 7f7ee9943640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ee40d1b20 0x7f7ee40ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.482+0000 7f7eea945640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee415e790 0x7f7ee40ca720 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400ca40 tx=0x7f7ed400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.482+0000 7f7ee2ffd640 1 -- 172.21.15.12:0/2565091919 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed400abd0 con 0x7f7ee415e790 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.482+0000 7f7ee2ffd640 1 -- 172.21.15.12:0/2565091919 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed400adb0 con 0x7f7ee415e790 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.482+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee40c9710 con 0x7f7ee415e790 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.482+0000 7f7ee2ffd640 1 -- 172.21.15.12:0/2565091919 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4012640 con 0x7f7ee415e790 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.482+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee414d7f0 con 0x7f7ee415e790 2024-04-15T15:22:14.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.483+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee40689e0 con 0x7f7ee415e790 2024-04-15T15:22:14.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.483+0000 7f7ee2ffd640 1 -- 172.21.15.12:0/2565091919 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ed40127e0 con 0x7f7ee415e790 2024-04-15T15:22:14.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.484+0000 7f7ee2ffd640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ed006c920 0x7f7ed006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.484+0000 7f7ee2ffd640 1 -- 172.21.15.12:0/2565091919 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f7ed408c520 con 0x7f7ee415e790 2024-04-15T15:22:14.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.484+0000 7f7eea144640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ed006c920 0x7f7ed006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.484+0000 7f7eea144640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ed006c920 0x7f7ed006ede0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00b840 tx=0x7f7edc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.488+0000 7f7ee2ffd640 1 -- 172.21.15.12:0/2565091919 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ed405a5a0 con 0x7f7ee415e790 2024-04-15T15:22:14.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.674+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7ee40d2f40 con 0x7f7ee415e790 2024-04-15T15:22:14.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.675+0000 7f7ee2ffd640 1 -- 172.21.15.12:0/2565091919 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f7ed4059ea0 con 0x7f7ee415e790 2024-04-15T15:22:14.676 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ed006c920 msgr2=0x7f7ed006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ed006c920 0x7f7ed006ede0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00b840 tx=0x7f7edc008040 comp rx=0 tx=0).stop 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee415e790 msgr2=0x7f7ee40ca720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee415e790 0x7f7ee40ca720 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400ca40 tx=0x7f7ed400cf00 comp rx=0 tx=0).stop 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 shutdown_connections 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee415e790 0x7f7ee40ca720 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7ed006c920 0x7f7ed006ede0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ee40d1b20 0x7f7ee40ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 --2- 172.21.15.12:0/2565091919 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7ee40b7be0 0x7f7ee414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 >> 172.21.15.12:0/2565091919 conn(0x7f7ee4067b40 msgr2=0x7f7ee415f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 shutdown_connections 2024-04-15T15:22:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.677+0000 7f7eec3a8640 1 -- 172.21.15.12:0/2565091919 wait complete. 2024-04-15T15:22:14.689 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.795+0000 7f8458b24640 1 Processor -- start 2024-04-15T15:22:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.795+0000 7f8458b24640 1 -- start start 2024-04-15T15:22:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.796+0000 7f8458b24640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.796+0000 7f8458b24640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f845415eae0 0x7f84540d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.796+0000 7f8458b24640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84540b47a0 con 0x7f845415e700 2024-04-15T15:22:14.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.796+0000 7f8458b24640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f84540b53b0 con 0x7f845415eae0 2024-04-15T15:22:14.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.796+0000 7f8458b24640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f84540b4a80 con 0x7f84540d2360 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f8451d9b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f8452d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f845415eae0 0x7f84540d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f845259c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f845415e700 0x7f84540b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:50130/0 (socket says 172.21.15.12:50130) 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f845259c640 1 -- 172.21.15.12:0/2895945110 learned_addr learned my addr 172.21.15.12:0/2895945110 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f8452d9d640 1 -- 172.21.15.12:0/2895945110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 msgr2=0x7f84540d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f8452d9d640 1 --2- 172.21.15.12:0/2895945110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f8452d9d640 1 -- 172.21.15.12:0/2895945110 >> v1:172.21.15.55:6789/0 conn(0x7f845415e700 legacy=0x7f84540b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f8452d9d640 1 -- 172.21.15.12:0/2895945110 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84540b4d60 con 0x7f845415eae0 2024-04-15T15:22:14.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.797+0000 7f8451d9b640 1 --2- 172.21.15.12:0/2895945110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:14.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.798+0000 7f8452d9d640 1 --2- 172.21.15.12:0/2895945110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 0x7f84540d30a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8444009d90 tx=0x7f844402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be480390daa70b49 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.798+0000 7f845159a640 1 -- 172.21.15.12:0/2895945110 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8444039070 con 0x7f845415eae0 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.798+0000 7f845159a640 1 -- 172.21.15.12:0/2895945110 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f844402cc50 con 0x7f845415eae0 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.798+0000 7f845159a640 1 -- 172.21.15.12:0/2895945110 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8444034440 con 0x7f845415eae0 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.799+0000 7f8458b24640 1 -- 172.21.15.12:0/2895945110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 msgr2=0x7f84540d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.799+0000 7f8458b24640 1 --2- 172.21.15.12:0/2895945110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 0x7f84540d30a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8444009d90 tx=0x7f844402bea0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.799+0000 7f8458b24640 1 -- 172.21.15.12:0/2895945110 shutdown_connections 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.799+0000 7f8458b24640 1 --2- 172.21.15.12:0/2895945110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 0x7f84540d30a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.799+0000 7f8458b24640 1 --2- 172.21.15.12:0/2895945110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.799+0000 7f8458b24640 1 -- 172.21.15.12:0/2895945110 >> 172.21.15.12:0/2895945110 conn(0x7f845408bf60 msgr2=0x7f84540d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:14.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.799+0000 7f8458b24640 1 -- 172.21.15.12:0/2895945110 shutdown_connections 2024-04-15T15:22:14.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.799+0000 7f8458b24640 1 -- 172.21.15.12:0/2895945110 wait complete. 2024-04-15T15:22:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.800+0000 7f8458b24640 1 Processor -- start 2024-04-15T15:22:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.800+0000 7f8458b24640 1 -- start start 2024-04-15T15:22:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8458b24640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540b9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8458b24640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f845415e700 0x7f845406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8458b24640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 0x7f845406e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8458b24640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f84540d1d40 con 0x7f845415e700 2024-04-15T15:22:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8458b24640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f84540b4450 con 0x7f845415eae0 2024-04-15T15:22:14.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8458b24640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f84540d1ec0 con 0x7f84540d2360 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8451d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f845415e700 0x7f845406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8451d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f845415e700 0x7f845406dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55570/0 (socket says 172.21.15.12:55570) 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8451d9b640 1 -- 172.21.15.12:0/1903628364 learned_addr learned my addr 172.21.15.12:0/1903628364 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f845259c640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540b9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8452d9d640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 0x7f845406e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8451d9b640 1 -- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 msgr2=0x7f84540b9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8451d9b640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540b9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8451d9b640 1 -- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 msgr2=0x7f845406e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8451d9b640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 0x7f845406e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.801+0000 7f8451d9b640 1 -- 172.21.15.12:0/1903628364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84541f8e60 con 0x7f845415e700 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.802+0000 7f8451d9b640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f845415e700 0x7f845406dcd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ec10 tx=0x7f843c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.802+0000 7f844affd640 1 -- 172.21.15.12:0/1903628364 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c017070 con 0x7f845415e700 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.802+0000 7f844affd640 1 -- 172.21.15.12:0/1903628364 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f843c00cc30 con 0x7f845415e700 2024-04-15T15:22:14.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.802+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84541fcf90 con 0x7f845415e700 2024-04-15T15:22:14.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.802+0000 7f844affd640 1 -- 172.21.15.12:0/1903628364 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c013470 con 0x7f845415e700 2024-04-15T15:22:14.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.802+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84541f9400 con 0x7f845415e700 2024-04-15T15:22:14.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.803+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84541f8c70 con 0x7f845415e700 2024-04-15T15:22:14.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.803+0000 7f844affd640 1 -- 172.21.15.12:0/1903628364 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f843c01a070 con 0x7f845415e700 2024-04-15T15:22:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.804+0000 7f844affd640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f843406c8f0 0x7f843406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.804+0000 7f844affd640 1 -- 172.21.15.12:0/1903628364 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f843c08bf00 con 0x7f845415e700 2024-04-15T15:22:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.804+0000 7f845259c640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f843406c8f0 0x7f843406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:14.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.804+0000 7f845259c640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f843406c8f0 0x7f843406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f844c0058d0 tx=0x7f844c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.808+0000 7f844affd640 1 -- 172.21.15.12:0/1903628364 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f843c059810 con 0x7f845415e700 2024-04-15T15:22:14.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.992+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84540689e0 con 0x7f845415e700 2024-04-15T15:22:14.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.992+0000 7f844affd640 1 -- 172.21.15.12:0/1903628364 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f843c05db10 con 0x7f845415e700 2024-04-15T15:22:14.994 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f843406c8f0 msgr2=0x7f843406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f843406c8f0 0x7f843406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f844c0058d0 tx=0x7f844c009040 comp rx=0 tx=0).stop 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f845415e700 msgr2=0x7f845406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f845415e700 0x7f845406dcd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ec10 tx=0x7f843c00c520 comp rx=0 tx=0).stop 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 shutdown_connections 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f843406c8f0 0x7f843406edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f845415eae0 0x7f845406e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f845415e700 0x7f845406dcd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 --2- 172.21.15.12:0/1903628364 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f84540d2360 0x7f84540b9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 >> 172.21.15.12:0/1903628364 conn(0x7f845408bf60 msgr2=0x7f84540b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 shutdown_connections 2024-04-15T15:22:14.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:14.994+0000 7f8458b24640 1 -- 172.21.15.12:0/1903628364 wait complete. 2024-04-15T15:22:15.006 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:15.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.112+0000 7f67fb91c640 1 Processor -- start 2024-04-15T15:22:15.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.112+0000 7f67fb91c640 1 -- start start 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67fb91c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67f415e830 0x7f67f40b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67fb91c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 0x7f67f40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67fb91c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f4067e30 con 0x7f67f415ec10 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67fb91c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f67f40b3c10 con 0x7f67f415e830 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67fb91c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f67f40b3ef0 con 0x7f67f40b53c0 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67f96b8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67f415e830 0x7f67f40b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67f8eb7640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f67f415ec10 0x7f67f4067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50878/0 (socket says 172.21.15.12:50878) 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67f8eb7640 1 -- 172.21.15.12:0/3275299961 learned_addr learned my addr 172.21.15.12:0/3275299961 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67f96b8640 1 -- 172.21.15.12:0/3275299961 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 msgr2=0x7f67f40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67f96b8640 1 --2- 172.21.15.12:0/3275299961 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 0x7f67f40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67f96b8640 1 -- 172.21.15.12:0/3275299961 >> v1:172.21.15.12:6789/0 conn(0x7f67f415ec10 legacy=0x7f67f4067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67ebfff640 1 -- 172.21.15.12:0/3275299961 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1159004189 0 0) 0x7f67f4067e30 con 0x7f67f415ec10 2024-04-15T15:22:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.113+0000 7f67f96b8640 1 -- 172.21.15.12:0/3275299961 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f40b41d0 con 0x7f67f415e830 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.114+0000 7f67f96b8640 1 --2- 172.21.15.12:0/3275299961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f415e830 0x7f67f40b4cb0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f67f00097a0 tx=0x7f67f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476be274da126308 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.114+0000 7f67ebfff640 1 -- 172.21.15.12:0/3275299961 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0035020 con 0x7f67f415e830 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.114+0000 7f67ebfff640 1 -- 172.21.15.12:0/3275299961 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f0031830 con 0x7f67f415e830 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.114+0000 7f67ebfff640 1 -- 172.21.15.12:0/3275299961 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0031c00 con 0x7f67f415e830 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.115+0000 7f67fb91c640 1 -- 172.21.15.12:0/3275299961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f415e830 msgr2=0x7f67f40b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.115+0000 7f67fb91c640 1 --2- 172.21.15.12:0/3275299961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f415e830 0x7f67f40b4cb0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f67f00097a0 tx=0x7f67f002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.115+0000 7f67fb91c640 1 -- 172.21.15.12:0/3275299961 shutdown_connections 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.115+0000 7f67fb91c640 1 --2- 172.21.15.12:0/3275299961 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 0x7f67f40b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.115+0000 7f67fb91c640 1 --2- 172.21.15.12:0/3275299961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f415e830 0x7f67f40b4cb0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.115+0000 7f67fb91c640 1 -- 172.21.15.12:0/3275299961 >> 172.21.15.12:0/3275299961 conn(0x7f67f4073360 msgr2=0x7f67f406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.115+0000 7f67fb91c640 1 -- 172.21.15.12:0/3275299961 shutdown_connections 2024-04-15T15:22:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.115+0000 7f67fb91c640 1 -- 172.21.15.12:0/3275299961 wait complete. 2024-04-15T15:22:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.116+0000 7f67fb91c640 1 Processor -- start 2024-04-15T15:22:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.116+0000 7f67fb91c640 1 -- start start 2024-04-15T15:22:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67fb91c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 0x7f67f41e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67fb91c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67f415e830 0x7f67f4147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67fb91c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40c6c40 0x7f67f4148010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67fb91c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f67f415ff60 con 0x7f67f40c6c40 2024-04-15T15:22:15.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67fb91c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f67f40b4790 con 0x7f67f415e830 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67fb91c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f67f41600e0 con 0x7f67f40b53c0 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40c6c40 0x7f67f4148010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40c6c40 0x7f67f4148010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55590/0 (socket says 172.21.15.12:55590) 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 -- 172.21.15.12:0/583991406 learned_addr learned my addr 172.21.15.12:0/583991406 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f96b8640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 0x7f67f41e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f8eb7640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67f415e830 0x7f67f4147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 -- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 msgr2=0x7f67f41e78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 0x7f67f41e78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 -- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67f415e830 msgr2=0x7f67f4147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67f415e830 0x7f67f4147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 -- 172.21.15.12:0/583991406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f41e9300 con 0x7f67f40c6c40 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f8eb7640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67f415e830 0x7f67f4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.117+0000 7f67f9eb9640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40c6c40 0x7f67f4148010 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00b970 tx=0x7f67ec00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.118+0000 7f67e9ffb640 1 -- 172.21.15.12:0/583991406 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec019070 con 0x7f67f40c6c40 2024-04-15T15:22:15.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.118+0000 7f67e9ffb640 1 -- 172.21.15.12:0/583991406 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ec00cb60 con 0x7f67f40c6c40 2024-04-15T15:22:15.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.118+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f41e6dd0 con 0x7f67f40c6c40 2024-04-15T15:22:15.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.118+0000 7f67e9ffb640 1 -- 172.21.15.12:0/583991406 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec014440 con 0x7f67f40c6c40 2024-04-15T15:22:15.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.118+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f41e7390 con 0x7f67f40c6c40 2024-04-15T15:22:15.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.119+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f40689e0 con 0x7f67f40c6c40 2024-04-15T15:22:15.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.119+0000 7f67e9ffb640 1 -- 172.21.15.12:0/583991406 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67ec01c070 con 0x7f67f40c6c40 2024-04-15T15:22:15.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.120+0000 7f67e9ffb640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67d806c920 0x7f67d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.120+0000 7f67e9ffb640 1 -- 172.21.15.12:0/583991406 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f67ec08bcd0 con 0x7f67f40c6c40 2024-04-15T15:22:15.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.120+0000 7f67f96b8640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67d806c920 0x7f67d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.121+0000 7f67f96b8640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67d806c920 0x7f67d806ede0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f67f0036000 tx=0x7f67f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.124+0000 7f67e9ffb640 1 -- 172.21.15.12:0/583991406 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67ec059cb0 con 0x7f67f40c6c40 2024-04-15T15:22:15.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.310+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f67f406ff10 con 0x7f67f40c6c40 2024-04-15T15:22:15.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.311+0000 7f67e9ffb640 1 -- 172.21.15.12:0/583991406 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f67ec0595b0 con 0x7f67f40c6c40 2024-04-15T15:22:15.312 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67d806c920 msgr2=0x7f67d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67d806c920 0x7f67d806ede0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f67f0036000 tx=0x7f67f0008030 comp rx=0 tx=0).stop 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40c6c40 msgr2=0x7f67f4148010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40c6c40 0x7f67f4148010 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00b970 tx=0x7f67ec00be30 comp rx=0 tx=0).stop 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 shutdown_connections 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67f40c6c40 0x7f67f4148010 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67d806c920 0x7f67d806ede0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67f415e830 0x7f67f4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 --2- 172.21.15.12:0/583991406 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67f40b53c0 0x7f67f41e78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 >> 172.21.15.12:0/583991406 conn(0x7f67f4073360 msgr2=0x7f67f40b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 shutdown_connections 2024-04-15T15:22:15.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.313+0000 7f67fb91c640 1 -- 172.21.15.12:0/583991406 wait complete. 2024-04-15T15:22:15.326 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:15.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.429+0000 7f125c9a8640 1 Processor -- start 2024-04-15T15:22:15.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.429+0000 7f125c9a8640 1 -- start start 2024-04-15T15:22:15.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f125c9a8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f125815eb00 0x7f12580ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f125c9a8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f12580b0560 0x7f125806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f125c9a8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12580cbd10 con 0x7f12580b09a0 2024-04-15T15:22:15.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f125c9a8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f125815eee0 con 0x7f125815eb00 2024-04-15T15:22:15.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f125c9a8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f12580cbff0 con 0x7f12580b0560 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f125659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f125815eb00 0x7f12580ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f125659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f125815eb00 0x7f12580ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55596/0 (socket says 172.21.15.12:55596) 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f125659c640 1 -- 172.21.15.12:0/1892802643 learned_addr learned my addr 172.21.15.12:0/1892802643 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.430+0000 7f1256d9d640 1 --1- 172.21.15.12:0/1892802643 >> v1:172.21.15.12:6789/0 conn(0x7f12580b09a0 0x7f12580ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50888/0 (socket says 172.21.15.12:50888) 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.431+0000 7f1255d9b640 1 --2- 172.21.15.12:0/1892802643 >> v2:172.21.15.55:3300/0 conn(0x7f12580b0560 0x7f125806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.431+0000 7f125659c640 1 -- 172.21.15.12:0/1892802643 >> v2:172.21.15.55:3300/0 conn(0x7f12580b0560 msgr2=0x7f125806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.431+0000 7f125659c640 1 --2- 172.21.15.12:0/1892802643 >> v2:172.21.15.55:3300/0 conn(0x7f12580b0560 0x7f125806b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.431+0000 7f125659c640 1 -- 172.21.15.12:0/1892802643 >> v1:172.21.15.12:6789/0 conn(0x7f12580b09a0 legacy=0x7f12580ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.431+0000 7f125559a640 1 -- 172.21.15.12:0/1892802643 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870482677 0 0) 0x7f12580cbd10 con 0x7f12580b09a0 2024-04-15T15:22:15.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.431+0000 7f125659c640 1 -- 172.21.15.12:0/1892802643 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12580cc2d0 con 0x7f125815eb00 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.431+0000 7f125659c640 1 --2- 172.21.15.12:0/1892802643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f125815eb00 0x7f12580ae2a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f12580cf0b0 tx=0x7f125002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad681e046782c0c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125559a640 1 -- 172.21.15.12:0/1892802643 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250034070 con 0x7f125815eb00 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125559a640 1 -- 172.21.15.12:0/1892802643 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1250030860 con 0x7f125815eb00 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125559a640 1 -- 172.21.15.12:0/1892802643 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250030c30 con 0x7f125815eb00 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125c9a8640 1 -- 172.21.15.12:0/1892802643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f125815eb00 msgr2=0x7f12580ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1892802643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f125815eb00 0x7f12580ae2a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f12580cf0b0 tx=0x7f125002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125c9a8640 1 -- 172.21.15.12:0/1892802643 shutdown_connections 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1892802643 >> v2:172.21.15.55:3300/0 conn(0x7f12580b0560 0x7f125806b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1892802643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f125815eb00 0x7f12580ae2a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125c9a8640 1 -- 172.21.15.12:0/1892802643 >> 172.21.15.12:0/1892802643 conn(0x7f125809a9d0 msgr2=0x7f12580d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125c9a8640 1 -- 172.21.15.12:0/1892802643 shutdown_connections 2024-04-15T15:22:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.432+0000 7f125c9a8640 1 -- 172.21.15.12:0/1892802643 wait complete. 2024-04-15T15:22:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f125c9a8640 1 Processor -- start 2024-04-15T15:22:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f125c9a8640 1 -- start start 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f125c9a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12580b0560 0x7f1258157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f125c9a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12580b09a0 0x7f12580c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f125c9a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f125815eb00 0x7f12580b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f125c9a8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12580ae7e0 con 0x7f12580b09a0 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f125c9a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f12580cc890 con 0x7f125815eb00 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f125c9a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f12580cec60 con 0x7f12580b0560 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f1255d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12580b09a0 0x7f12580c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f1255d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12580b09a0 0x7f12580c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55598/0 (socket says 172.21.15.12:55598) 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.434+0000 7f1255d9b640 1 -- 172.21.15.12:0/1345158415 learned_addr learned my addr 172.21.15.12:0/1345158415 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f125659c640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12580b0560 0x7f1258157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f1256d9d640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f125815eb00 0x7f12580b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f1255d9b640 1 -- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12580b0560 msgr2=0x7f1258157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f1255d9b640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12580b0560 0x7f1258157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f1255d9b640 1 -- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f125815eb00 msgr2=0x7f12580b6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f1255d9b640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f125815eb00 0x7f12580b6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f1255d9b640 1 -- 172.21.15.12:0/1345158415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12580c94e0 con 0x7f12580b09a0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f1256d9d640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f125815eb00 0x7f12580b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f1255d9b640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12580b09a0 0x7f12580c6fb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f124000d800 tx=0x7f124000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f124effd640 1 -- 172.21.15.12:0/1345158415 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124000b6c0 con 0x7f12580b09a0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f124effd640 1 -- 172.21.15.12:0/1345158415 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1240019070 con 0x7f12580b09a0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12580c97c0 con 0x7f12580b09a0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f124effd640 1 -- 172.21.15.12:0/1345158415 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124000ba80 con 0x7f12580b09a0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.435+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12580c9d80 con 0x7f12580b09a0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.436+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12580690f0 con 0x7f12580b09a0 2024-04-15T15:22:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.437+0000 7f124effd640 1 -- 172.21.15.12:0/1345158415 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f124000bc20 con 0x7f12580b09a0 2024-04-15T15:22:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.437+0000 7f124effd640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f123806c8f0 0x7f123806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.437+0000 7f124effd640 1 -- 172.21.15.12:0/1345158415 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f124008c390 con 0x7f12580b09a0 2024-04-15T15:22:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.437+0000 7f125659c640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f123806c8f0 0x7f123806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.438+0000 7f125659c640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f123806c8f0 0x7f123806edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1250036000 tx=0x7f125003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.441+0000 7f124effd640 1 -- 172.21.15.12:0/1345158415 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f124005a410 con 0x7f12580b09a0 2024-04-15T15:22:15.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.637+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f125815e160 con 0x7f12580b09a0 2024-04-15T15:22:15.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.637+0000 7f124effd640 1 -- 172.21.15.12:0/1345158415 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f1240059d10 con 0x7f12580b09a0 2024-04-15T15:22:15.639 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:15.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f123806c8f0 msgr2=0x7f123806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f123806c8f0 0x7f123806edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1250036000 tx=0x7f125003b040 comp rx=0 tx=0).stop 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12580b09a0 msgr2=0x7f12580c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12580b09a0 0x7f12580c6fb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f124000d800 tx=0x7f124000dcc0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 shutdown_connections 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f123806c8f0 0x7f123806edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f125815eb00 0x7f12580b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12580b09a0 0x7f12580c6fb0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 --2- 172.21.15.12:0/1345158415 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12580b0560 0x7f1258157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 >> 172.21.15.12:0/1345158415 conn(0x7f125809a9d0 msgr2=0x7f12580b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 shutdown_connections 2024-04-15T15:22:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.639+0000 7f125c9a8640 1 -- 172.21.15.12:0/1345158415 wait complete. 2024-04-15T15:22:15.652 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:22:15.652 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:15.673 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.776+0000 7f55ee5d5640 1 Processor -- start 2024-04-15T15:22:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.776+0000 7f55ee5d5640 1 -- start start 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55ee5d5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 0x7f55e80d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55ee5d5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55e80d2480 con 0x7f55e80d1610 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55ee5d5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55e80d2760 con 0x7f55e80cfd20 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55ee5d5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f55e80d2db0 con 0x7f55e815e790 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55ecb72640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f55e80d1610 0x7f55e80b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50896/0 (socket says 172.21.15.12:50896) 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55ecb72640 1 -- 172.21.15.12:0/2152930295 learned_addr learned my addr 172.21.15.12:0/2152930295 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55e7fff640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 0x7f55e80d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3605348369 0 0) 0x7f55e80d2480 con 0x7f55e80d1610 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.777+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55d4003520 con 0x7f55e80d1610 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2717447396 0 0) 0x7f55d4003520 con 0x7f55e80d1610 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 msgr2=0x7f55e80d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 0x7f55e80d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 >> v1:172.21.15.55:6789/0 conn(0x7f55e80cfd20 legacy=0x7f55e80d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e80d2a40 con 0x7f55e80d1610 2024-04-15T15:22:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f55e00019f0 con 0x7f55e80d1610 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 >> v1:172.21.15.12:6789/0 conn(0x7f55e80d1610 legacy=0x7f55e80b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 0x7f55d40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55d4003f20 0x7f55d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4004d40 0x7f55d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f55e80d2db0 con 0x7f55d4003f20 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f55d400ba20 con 0x7f55d4004d40 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f55d400bc20 con 0x7f55e815e790 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55dffff640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55d4003f20 0x7f55d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e7fff640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 0x7f55d40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55ecb72640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4004d40 0x7f55d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55dffff640 1 -- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 msgr2=0x7f55d40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55dffff640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 0x7f55d40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55dffff640 1 -- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4004d40 msgr2=0x7f55d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55dffff640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4004d40 0x7f55d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55dffff640 1 -- 172.21.15.12:0/2152930295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d4003520 con 0x7f55d4003f20 2024-04-15T15:22:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.778+0000 7f55e7fff640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e815e790 0x7f55d40039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.779+0000 7f55dffff640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55d4003f20 0x7f55d4004800 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f55d0001860 tx=0x7f55d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd5d5b5022e3435 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.779+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d0001430 con 0x7f55d4003f20 2024-04-15T15:22:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.779+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d0032620 con 0x7f55d4003f20 2024-04-15T15:22:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.779+0000 7f55e77fe640 1 -- 172.21.15.12:0/2152930295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d0032a10 con 0x7f55d4003f20 2024-04-15T15:22:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.780+0000 7f55ee5d5640 1 -- 172.21.15.12:0/2152930295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55d4003f20 msgr2=0x7f55d4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.780+0000 7f55ee5d5640 1 --2- 172.21.15.12:0/2152930295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55d4003f20 0x7f55d4004800 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f55d0001860 tx=0x7f55d4005830 comp rx=0 tx=0).stop 2024-04-15T15:22:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.780+0000 7f55ecb72640 1 -- 172.21.15.12:0/2152930295 reap_dead start 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.780+0000 7f55ee5d5640 1 -- 172.21.15.12:0/2152930295 shutdown_connections 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.780+0000 7f55ee5d5640 1 -- 172.21.15.12:0/2152930295 >> 172.21.15.12:0/2152930295 conn(0x7f55e808bf60 msgr2=0x7f55e80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.780+0000 7f55ee5d5640 1 -- 172.21.15.12:0/2152930295 shutdown_connections 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.780+0000 7f55ee5d5640 1 -- 172.21.15.12:0/2152930295 wait complete. 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.781+0000 7f55ee5d5640 1 Processor -- start 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.781+0000 7f55ee5d5640 1 -- start start 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ee5d5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4003f20 0x7f55e814ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ee5d5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e80ca250 0x7f55e806dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ee5d5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80ca6d0 0x7f55e806e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ee5d5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f55e815ecf0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ee5d5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f55e815eb70 con 0x7f55d4003f20 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ee5d5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f55e80d0160 con 0x7f55e80ca250 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80ca6d0 0x7f55e806e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80ca6d0 0x7f55e806e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55608/0 (socket says 172.21.15.12:55608) 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 -- 172.21.15.12:0/4139673485 learned_addr learned my addr 172.21.15.12:0/4139673485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55e7fff640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4003f20 0x7f55e814ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55dffff640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e80ca250 0x7f55e806dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 -- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e80ca250 msgr2=0x7f55e806dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e80ca250 0x7f55e806dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 -- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4003f20 msgr2=0x7f55e814ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4003f20 0x7f55e814ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 -- 172.21.15.12:0/4139673485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e815fcf0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55e7fff640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4003f20 0x7f55e814ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55dffff640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e80ca250 0x7f55e806dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ecb72640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80ca6d0 0x7f55e806e1e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f55e808d280 tx=0x7f55e0007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55e57fa640 1 -- 172.21.15.12:0/4139673485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e000ab60 con 0x7f55e80ca6d0 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55e57fa640 1 -- 172.21.15.12:0/4139673485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55e0019070 con 0x7f55e80ca6d0 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.782+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e815ffd0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.783+0000 7f55e57fa640 1 -- 172.21.15.12:0/4139673485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e00133f0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.783+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e8160590 con 0x7f55e80ca6d0 2024-04-15T15:22:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.784+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55e80689e0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.784+0000 7f55e57fa640 1 -- 172.21.15.12:0/4139673485 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55e0013590 con 0x7f55e80ca6d0 2024-04-15T15:22:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.784+0000 7f55e57fa640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f55c806c920 0x7f55c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.784+0000 7f55e57fa640 1 -- 172.21.15.12:0/4139673485 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f55e008c4c0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.785+0000 7f55e7fff640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f55c806c920 0x7f55c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:15.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.785+0000 7f55e7fff640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f55c806c920 0x7f55c806ede0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009a80 tx=0x7f55d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.789+0000 7f55e57fa640 1 -- 172.21.15.12:0/4139673485 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55e005a4b0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.975+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55e80d1ce0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.976+0000 7f55e57fa640 1 -- 172.21.15.12:0/4139673485 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f55e0059db0 con 0x7f55e80ca6d0 2024-04-15T15:22:15.978 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:15.978 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f55c806c920 msgr2=0x7f55c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f55c806c920 0x7f55c806ede0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009a80 tx=0x7f55d8008040 comp rx=0 tx=0).stop 2024-04-15T15:22:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80ca6d0 msgr2=0x7f55e806e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80ca6d0 0x7f55e806e1e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f55e808d280 tx=0x7f55e0007a90 comp rx=0 tx=0).stop 2024-04-15T15:22:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 shutdown_connections 2024-04-15T15:22:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80ca6d0 0x7f55e806e1e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f55e80ca250 0x7f55e806dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f55c806c920 0x7f55c806ede0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 --2- 172.21.15.12:0/4139673485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55d4003f20 0x7f55e814ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 >> 172.21.15.12:0/4139673485 conn(0x7f55e808bf60 msgr2=0x7f55e80d3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 shutdown_connections 2024-04-15T15:22:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:15.978+0000 7f55ee5d5640 1 -- 172.21.15.12:0/4139673485 wait complete. 2024-04-15T15:22:15.991 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:15.992 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:16.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.097+0000 7f7aa5b32640 1 Processor -- start 2024-04-15T15:22:16.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.097+0000 7f7aa5b32640 1 -- start start 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.098+0000 7f7aa5b32640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7aa015d160 0x7f7aa0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.098+0000 7f7aa5b32640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7aa006e6f0 0x7f7aa0163a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.098+0000 7f7aa5b32640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa00d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.098+0000 7f7aa5b32640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7aa0072a00 con 0x7f7aa015d160 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.098+0000 7f7aa5b32640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7aa015d540 con 0x7f7aa006e6f0 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.098+0000 7f7aa5b32640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7aa00d0ff0 con 0x7f7aa00d0690 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.098+0000 7f7a9f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7aa015d160 0x7f7aa0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7aa015d160 0x7f7aa0071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55610/0 (socket says 172.21.15.12:55610) 2024-04-15T15:22:16.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9f7fe640 1 -- 172.21.15.12:0/2657446354 learned_addr learned my addr 172.21.15.12:0/2657446354 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9ffff640 1 --2- 172.21.15.12:0/2657446354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa00d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9effd640 1 --2- 172.21.15.12:0/2657446354 >> v2:172.21.15.55:3300/0 conn(0x7f7aa006e6f0 0x7f7aa0163a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9f7fe640 1 -- 172.21.15.12:0/2657446354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 msgr2=0x7f7aa00d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9f7fe640 1 --2- 172.21.15.12:0/2657446354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa00d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9f7fe640 1 -- 172.21.15.12:0/2657446354 >> v2:172.21.15.55:3300/0 conn(0x7f7aa006e6f0 msgr2=0x7f7aa0163a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9f7fe640 1 --2- 172.21.15.12:0/2657446354 >> v2:172.21.15.55:3300/0 conn(0x7f7aa006e6f0 0x7f7aa0163a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9f7fe640 1 -- 172.21.15.12:0/2657446354 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa0072440 con 0x7f7aa015d160 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9effd640 1 --2- 172.21.15.12:0/2657446354 >> v2:172.21.15.55:3300/0 conn(0x7f7aa006e6f0 0x7f7aa0163a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.099+0000 7f7a9ffff640 1 --2- 172.21.15.12:0/2657446354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa00d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7a9f7fe640 1 --2- 172.21.15.12:0/2657446354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa015d160 0x7f7aa0071d10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7aa006fe90 tx=0x7f7a9002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9599dd3cfe433b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7a9e7fc640 1 -- 172.21.15.12:0/2657446354 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a90035020 con 0x7f7aa015d160 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7a9e7fc640 1 -- 172.21.15.12:0/2657446354 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a90031640 con 0x7f7aa015d160 2024-04-15T15:22:16.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7a9e7fc640 1 -- 172.21.15.12:0/2657446354 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9002d040 con 0x7f7aa015d160 2024-04-15T15:22:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2657446354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa015d160 msgr2=0x7f7aa0071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2657446354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa015d160 0x7f7aa0071d10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7aa006fe90 tx=0x7f7a9002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2657446354 shutdown_connections 2024-04-15T15:22:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2657446354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa00d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2657446354 >> v2:172.21.15.55:3300/0 conn(0x7f7aa006e6f0 0x7f7aa0163a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2657446354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa015d160 0x7f7aa0071d10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2657446354 >> 172.21.15.12:0/2657446354 conn(0x7f7aa006be00 msgr2=0x7f7aa006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:16.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.100+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2657446354 shutdown_connections 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.101+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2657446354 wait complete. 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.102+0000 7f7aa5b32640 1 Processor -- start 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.102+0000 7f7aa5b32640 1 -- start start 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7aa5b32640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa006e6f0 0x7f7aa0160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7aa5b32640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa015bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7aa5b32640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7aa015d160 0x7f7aa00ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7aa5b32640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa006e1d0 con 0x7f7aa006e6f0 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7aa5b32640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa006def0 con 0x7f7aa015d160 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7aa5b32640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7aa0163fa0 con 0x7f7aa00d0690 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa006e6f0 0x7f7aa0160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa006e6f0 0x7f7aa0160240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55618/0 (socket says 172.21.15.12:55618) 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 -- 172.21.15.12:0/2664917843 learned_addr learned my addr 172.21.15.12:0/2664917843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:16.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9ffff640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7aa015d160 0x7f7aa00ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9effd640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa015bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 -- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 msgr2=0x7f7aa015bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa015bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 -- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7aa015d160 msgr2=0x7f7aa00ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7aa015d160 0x7f7aa00ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 -- 172.21.15.12:0/2664917843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa015c3e0 con 0x7f7aa006e6f0 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9effd640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa015bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9ffff640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7aa015d160 0x7f7aa00ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a9f7fe640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa006e6f0 0x7f7aa0160240 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7a90000c00 tx=0x7f7a900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a87fff640 1 -- 172.21.15.12:0/2664917843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a90040070 con 0x7f7aa006e6f0 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a87fff640 1 -- 172.21.15.12:0/2664917843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a900399f0 con 0x7f7aa006e6f0 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa015f1a0 con 0x7f7aa006e6f0 2024-04-15T15:22:16.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7a87fff640 1 -- 172.21.15.12:0/2664917843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a90035040 con 0x7f7aa006e6f0 2024-04-15T15:22:16.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.103+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa015f760 con 0x7f7aa006e6f0 2024-04-15T15:22:16.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.104+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa00689e0 con 0x7f7aa006e6f0 2024-04-15T15:22:16.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.105+0000 7f7a87fff640 1 -- 172.21.15.12:0/2664917843 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a9004b060 con 0x7f7aa006e6f0 2024-04-15T15:22:16.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.105+0000 7f7a87fff640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7a8c06c8f0 0x7f7a8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.105+0000 7f7a87fff640 1 -- 172.21.15.12:0/2664917843 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f7a900ad8a0 con 0x7f7aa006e6f0 2024-04-15T15:22:16.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.106+0000 7f7a9effd640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7a8c06c8f0 0x7f7a8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.106+0000 7f7a9effd640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7a8c06c8f0 0x7f7a8c06edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7aa01fbf50 tx=0x7f7a94008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.110+0000 7f7a87fff640 1 -- 172.21.15.12:0/2664917843 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7aa015c3e0 con 0x7f7aa006e6f0 2024-04-15T15:22:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.293+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7aa006f9e0 con 0x7f7aa006e6f0 2024-04-15T15:22:16.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.294+0000 7f7a87fff640 1 -- 172.21.15.12:0/2664917843 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f7a9007b890 con 0x7f7aa006e6f0 2024-04-15T15:22:16.363 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7a8c06c8f0 msgr2=0x7f7a8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7a8c06c8f0 0x7f7a8c06edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7aa01fbf50 tx=0x7f7a94008040 comp rx=0 tx=0).stop 2024-04-15T15:22:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa006e6f0 msgr2=0x7f7aa0160240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa006e6f0 0x7f7aa0160240 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7a90000c00 tx=0x7f7a900046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 shutdown_connections 2024-04-15T15:22:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7a8c06c8f0 0x7f7a8c06edb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7aa015d160 0x7f7aa00ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7aa00d0690 0x7f7aa015bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 --2- 172.21.15.12:0/2664917843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7aa006e6f0 0x7f7aa0160240 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 >> 172.21.15.12:0/2664917843 conn(0x7f7aa006be00 msgr2=0x7f7aa00b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 shutdown_connections 2024-04-15T15:22:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.296+0000 7f7aa5b32640 1 -- 172.21.15.12:0/2664917843 wait complete. 2024-04-15T15:22:16.365 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:16.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.472+0000 7f12fb847640 1 Processor -- start 2024-04-15T15:22:16.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.472+0000 7f12fb847640 1 -- start start 2024-04-15T15:22:16.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12fb847640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f415e790 0x7f12f40d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12fb847640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f12f40cfd20 0x7f12f40d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12fb847640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f40d2480 con 0x7f12f40d0f00 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12fb847640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f12f40d2db0 con 0x7f12f40cfd20 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12fb847640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f12f40d2760 con 0x7f12f415e790 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12f8de2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f12f40cfd20 0x7f12f40d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12f8de2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f12f40cfd20 0x7f12f40d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55634/0 (socket says 172.21.15.12:55634) 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12f8de2640 1 -- 172.21.15.12:0/775920835 learned_addr learned my addr 172.21.15.12:0/775920835 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12f95e3640 1 --2- 172.21.15.12:0/775920835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f415e790 0x7f12f40d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12f8de2640 1 -- 172.21.15.12:0/775920835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f415e790 msgr2=0x7f12f40d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12f8de2640 1 --2- 172.21.15.12:0/775920835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f415e790 0x7f12f40d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12f8de2640 1 -- 172.21.15.12:0/775920835 >> v1:172.21.15.55:6789/0 conn(0x7f12f40d0f00 legacy=0x7f12f40b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.473+0000 7f12f8de2640 1 -- 172.21.15.12:0/775920835 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f40d2a40 con 0x7f12f40cfd20 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.474+0000 7f12f8de2640 1 --2- 172.21.15.12:0/775920835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f40cfd20 0x7f12f40d0160 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009870 tx=0x7f12e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb98c6bde7e9fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.474+0000 7f12ebfff640 1 -- 172.21.15.12:0/775920835 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4034070 con 0x7f12f40cfd20 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.474+0000 7f12ebfff640 1 -- 172.21.15.12:0/775920835 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e4030840 con 0x7f12f40cfd20 2024-04-15T15:22:16.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.474+0000 7f12ebfff640 1 -- 172.21.15.12:0/775920835 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4030c10 con 0x7f12f40cfd20 2024-04-15T15:22:16.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.475+0000 7f12fb847640 1 -- 172.21.15.12:0/775920835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f40cfd20 msgr2=0x7f12f40d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.475+0000 7f12fb847640 1 --2- 172.21.15.12:0/775920835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f40cfd20 0x7f12f40d0160 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009870 tx=0x7f12e402b980 comp rx=0 tx=0).stop 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.475+0000 7f12fb847640 1 -- 172.21.15.12:0/775920835 shutdown_connections 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.475+0000 7f12fb847640 1 --2- 172.21.15.12:0/775920835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f40cfd20 0x7f12f40d0160 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.475+0000 7f12fb847640 1 --2- 172.21.15.12:0/775920835 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f415e790 0x7f12f40d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.475+0000 7f12fb847640 1 -- 172.21.15.12:0/775920835 >> 172.21.15.12:0/775920835 conn(0x7f12f408bf60 msgr2=0x7f12f40d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.475+0000 7f12fb847640 1 -- 172.21.15.12:0/775920835 shutdown_connections 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.475+0000 7f12fb847640 1 -- 172.21.15.12:0/775920835 wait complete. 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.476+0000 7f12fb847640 1 Processor -- start 2024-04-15T15:22:16.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.476+0000 7f12fb847640 1 -- start start 2024-04-15T15:22:16.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12fb847640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f40cfd20 0x7f12f40b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12fb847640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f40d0f00 0x7f12f4154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12fb847640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f415e790 0x7f12f40a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12fb847640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12f415ecf0 con 0x7f12f415e790 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12fb847640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f12f415eb70 con 0x7f12f40d0f00 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12fb847640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f12f40d13b0 con 0x7f12f40cfd20 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f415e790 0x7f12f40a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f415e790 0x7f12f40a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55642/0 (socket says 172.21.15.12:55642) 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 -- 172.21.15.12:0/2039120992 learned_addr learned my addr 172.21.15.12:0/2039120992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f95e3640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f40cfd20 0x7f12f40b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f8de2640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f40d0f00 0x7f12f4154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 -- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f40cfd20 msgr2=0x7f12f40b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f40cfd20 0x7f12f40b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 -- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f40d0f00 msgr2=0x7f12f4154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f40d0f00 0x7f12f4154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 -- 172.21.15.12:0/2039120992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f40a2430 con 0x7f12f415e790 2024-04-15T15:22:16.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.477+0000 7f12f9de4640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f415e790 0x7f12f40a1ef0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00c740 tx=0x7f12ec00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.478+0000 7f12e9ffb640 1 -- 172.21.15.12:0/2039120992 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec016020 con 0x7f12f415e790 2024-04-15T15:22:16.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.478+0000 7f12e9ffb640 1 -- 172.21.15.12:0/2039120992 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12ec00ce50 con 0x7f12f415e790 2024-04-15T15:22:16.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.478+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f40a0650 con 0x7f12f415e790 2024-04-15T15:22:16.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.478+0000 7f12e9ffb640 1 -- 172.21.15.12:0/2039120992 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec003970 con 0x7f12f415e790 2024-04-15T15:22:16.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.478+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f40a0c10 con 0x7f12f415e790 2024-04-15T15:22:16.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.479+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f40689e0 con 0x7f12f415e790 2024-04-15T15:22:16.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.479+0000 7f12e9ffb640 1 -- 172.21.15.12:0/2039120992 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f12ec003b10 con 0x7f12f415e790 2024-04-15T15:22:16.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.480+0000 7f12e9ffb640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f12d806c8f0 0x7f12d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.480+0000 7f12e9ffb640 1 -- 172.21.15.12:0/2039120992 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f12ec08bbb0 con 0x7f12f415e790 2024-04-15T15:22:16.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.480+0000 7f12f95e3640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f12d806c8f0 0x7f12d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.480+0000 7f12f95e3640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f12d806c8f0 0x7f12d806edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f12f00079a0 tx=0x7f12f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.484+0000 7f12e9ffb640 1 -- 172.21.15.12:0/2039120992 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f12ec020020 con 0x7f12f415e790 2024-04-15T15:22:16.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.666+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f12f40d1ce0 con 0x7f12f415e790 2024-04-15T15:22:16.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.667+0000 7f12e9ffb640 1 -- 172.21.15.12:0/2039120992 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f12ec0594b0 con 0x7f12f415e790 2024-04-15T15:22:16.668 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f12d806c8f0 msgr2=0x7f12d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f12d806c8f0 0x7f12d806edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f12f00079a0 tx=0x7f12f0008040 comp rx=0 tx=0).stop 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f415e790 msgr2=0x7f12f40a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f415e790 0x7f12f40a1ef0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00c740 tx=0x7f12ec00cc00 comp rx=0 tx=0).stop 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 shutdown_connections 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12f415e790 0x7f12f40a1ef0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f12d806c8f0 0x7f12d806edb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f40d0f00 0x7f12f4154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 --2- 172.21.15.12:0/2039120992 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f12f40cfd20 0x7f12f40b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 >> 172.21.15.12:0/2039120992 conn(0x7f12f408bf60 msgr2=0x7f12f40b8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 shutdown_connections 2024-04-15T15:22:16.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.669+0000 7f12fb847640 1 -- 172.21.15.12:0/2039120992 wait complete. 2024-04-15T15:22:16.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:16.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.788+0000 7fef530c5640 1 Processor -- start 2024-04-15T15:22:16.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.788+0000 7fef530c5640 1 -- start start 2024-04-15T15:22:16.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef530c5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c15e790 0x7fef4c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef530c5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef4c0d1610 0x7fef4c0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef530c5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef4c0d2480 con 0x7fef4c0cfd20 2024-04-15T15:22:16.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef530c5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fef4c0d2db0 con 0x7fef4c0d1610 2024-04-15T15:22:16.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef530c5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fef4c0d2760 con 0x7fef4c15e790 2024-04-15T15:22:16.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef51662640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef4c0d1610 0x7fef4c0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef51662640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef4c0d1610 0x7fef4c0d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55658/0 (socket says 172.21.15.12:55658) 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef51662640 1 -- 172.21.15.12:0/36039953 learned_addr learned my addr 172.21.15.12:0/36039953 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef51662640 1 -- 172.21.15.12:0/36039953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c15e790 msgr2=0x7fef4c0d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef50e61640 1 --2- 172.21.15.12:0/36039953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c15e790 0x7fef4c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef51662640 1 --2- 172.21.15.12:0/36039953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c15e790 0x7fef4c0d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef51662640 1 -- 172.21.15.12:0/36039953 >> v1:172.21.15.55:6789/0 conn(0x7fef4c0cfd20 legacy=0x7fef4c0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef51662640 1 -- 172.21.15.12:0/36039953 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c0d2a40 con 0x7fef4c0d1610 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.789+0000 7fef50e61640 1 --2- 172.21.15.12:0/36039953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c15e790 0x7fef4c0d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.790+0000 7fef51662640 1 --2- 172.21.15.12:0/36039953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c0d1610 0x7fef4c0d0160 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fef44009870 tx=0x7fef4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7675a902f381259f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.790+0000 7fef437fe640 1 -- 172.21.15.12:0/36039953 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44035020 con 0x7fef4c0d1610 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.790+0000 7fef437fe640 1 -- 172.21.15.12:0/36039953 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef44031640 con 0x7fef4c0d1610 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.790+0000 7fef437fe640 1 -- 172.21.15.12:0/36039953 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4402d040 con 0x7fef4c0d1610 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.790+0000 7fef530c5640 1 -- 172.21.15.12:0/36039953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c0d1610 msgr2=0x7fef4c0d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.790+0000 7fef530c5640 1 --2- 172.21.15.12:0/36039953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c0d1610 0x7fef4c0d0160 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fef44009870 tx=0x7fef4402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.791+0000 7fef530c5640 1 -- 172.21.15.12:0/36039953 shutdown_connections 2024-04-15T15:22:16.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.791+0000 7fef530c5640 1 --2- 172.21.15.12:0/36039953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c0d1610 0x7fef4c0d0160 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.791+0000 7fef530c5640 1 --2- 172.21.15.12:0/36039953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c15e790 0x7fef4c0d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.791+0000 7fef530c5640 1 -- 172.21.15.12:0/36039953 >> 172.21.15.12:0/36039953 conn(0x7fef4c08bf60 msgr2=0x7fef4c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:16.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.791+0000 7fef530c5640 1 -- 172.21.15.12:0/36039953 shutdown_connections 2024-04-15T15:22:16.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.791+0000 7fef530c5640 1 -- 172.21.15.12:0/36039953 wait complete. 2024-04-15T15:22:16.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.792+0000 7fef530c5640 1 Processor -- start 2024-04-15T15:22:16.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.792+0000 7fef530c5640 1 -- start start 2024-04-15T15:22:16.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef530c5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef4c0cfd20 0x7fef4c153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef530c5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c0d1610 0x7fef4c0cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef530c5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c15e790 0x7fef4c0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef530c5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c15ecf0 con 0x7fef4c15e790 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef530c5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c15eb70 con 0x7fef4c0cfd20 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef530c5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fef4c0b74f0 con 0x7fef4c0d1610 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c15e790 0x7fef4c0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c15e790 0x7fef4c0cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55670/0 (socket says 172.21.15.12:55670) 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 -- 172.21.15.12:0/3216954039 learned_addr learned my addr 172.21.15.12:0/3216954039 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef50e61640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef4c0cfd20 0x7fef4c153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef43fff640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c0d1610 0x7fef4c0cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 -- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c0d1610 msgr2=0x7fef4c0cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c0d1610 0x7fef4c0cd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 -- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef4c0cfd20 msgr2=0x7fef4c153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef4c0cfd20 0x7fef4c153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 -- 172.21.15.12:0/3216954039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c15d2b0 con 0x7fef4c15e790 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef50e61640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef4c0cfd20 0x7fef4c153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef51662640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c15e790 0x7fef4c0cde80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef4402b930 tx=0x7fef440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.793+0000 7fef417fa640 1 -- 172.21.15.12:0/3216954039 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44040070 con 0x7fef4c15e790 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.794+0000 7fef417fa640 1 -- 172.21.15.12:0/3216954039 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef44039cd0 con 0x7fef4c15e790 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.794+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4c0b1ed0 con 0x7fef4c15e790 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.794+0000 7fef417fa640 1 -- 172.21.15.12:0/3216954039 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44035040 con 0x7fef4c15e790 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.794+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef4c0b2770 con 0x7fef4c15e790 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.794+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4c0b22c0 con 0x7fef4c15e790 2024-04-15T15:22:16.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.795+0000 7fef417fa640 1 -- 172.21.15.12:0/3216954039 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef4404b060 con 0x7fef4c15e790 2024-04-15T15:22:16.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.795+0000 7fef417fa640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef3806c8f0 0x7fef3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:16.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.796+0000 7fef417fa640 1 -- 172.21.15.12:0/3216954039 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fef440ada30 con 0x7fef4c15e790 2024-04-15T15:22:16.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.796+0000 7fef50e61640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef3806c8f0 0x7fef3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:16.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.796+0000 7fef50e61640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef3806c8f0 0x7fef3806edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef48007f40 tx=0x7fef48007490 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:16.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.800+0000 7fef417fa640 1 -- 172.21.15.12:0/3216954039 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef4407bab0 con 0x7fef4c15e790 2024-04-15T15:22:16.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.986+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fef4c0689e0 con 0x7fef4c15e790 2024-04-15T15:22:16.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.988+0000 7fef417fa640 1 -- 172.21.15.12:0/3216954039 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fef4407b3b0 con 0x7fef4c15e790 2024-04-15T15:22:16.989 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef3806c8f0 msgr2=0x7fef3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef3806c8f0 0x7fef3806edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef48007f40 tx=0x7fef48007490 comp rx=0 tx=0).stop 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c15e790 msgr2=0x7fef4c0cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c15e790 0x7fef4c0cde80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef4402b930 tx=0x7fef440046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 shutdown_connections 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef4c15e790 0x7fef4c0cde80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fef4c0d1610 0x7fef4c0cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef3806c8f0 0x7fef3806edb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 --2- 172.21.15.12:0/3216954039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef4c0cfd20 0x7fef4c153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 >> 172.21.15.12:0/3216954039 conn(0x7fef4c08bf60 msgr2=0x7fef4c0b8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 shutdown_connections 2024-04-15T15:22:16.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:16.990+0000 7fef530c5640 1 -- 172.21.15.12:0/3216954039 wait complete. 2024-04-15T15:22:17.003 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.109+0000 7f4992f72640 1 Processor -- start 2024-04-15T15:22:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.109+0000 7f4992f72640 1 -- start start 2024-04-15T15:22:17.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f4992f72640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f498c09a9d0 0x7f498c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f4992f72640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 0x7f498c15fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f4992f72640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f498c070720 con 0x7f498c09adb0 2024-04-15T15:22:17.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f4992f72640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f498c09ba90 con 0x7f498c09a9d0 2024-04-15T15:22:17.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f4992f72640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f498c160070 con 0x7f498c0701f0 2024-04-15T15:22:17.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f499150f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f498c09adb0 0x7f498c0d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50908/0 (socket says 172.21.15.12:50908) 2024-04-15T15:22:17.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f499150f640 1 -- 172.21.15.12:0/806201925 learned_addr learned my addr 172.21.15.12:0/806201925 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f4990d0e640 1 --2- 172.21.15.12:0/806201925 >> v2:172.21.15.55:3300/0 conn(0x7f498c09a9d0 0x7f498c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498bfff640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 0x7f498c15fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3521312057 0 0) 0x7f498c070720 con 0x7f498c09adb0 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f497c003520 con 0x7f498c09adb0 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1821623693 0 0) 0x7f497c003520 con 0x7f498c09adb0 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 msgr2=0x7f498c15fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 0x7f498c15fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 >> v2:172.21.15.55:3300/0 conn(0x7f498c09a9d0 msgr2=0x7f498c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 --2- 172.21.15.12:0/806201925 >> v2:172.21.15.55:3300/0 conn(0x7f498c09a9d0 0x7f498c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498c070a90 con 0x7f498c09adb0 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f49840019f0 con 0x7f498c09adb0 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498bfff640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 0x7f498c15fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 >> v1:172.21.15.12:6789/0 conn(0x7f498c09adb0 legacy=0x7f498c0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0039b0 0x7f497c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c004800 0x7f497c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 0x7f497c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f498c160070 con 0x7f497c004800 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f497c00b9c0 con 0x7f497c0039b0 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.110+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f497c00bb40 con 0x7f498c0701f0 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.111+0000 7f498bfff640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c004800 0x7f497c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.111+0000 7f4990d0e640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0039b0 0x7f497c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.111+0000 7f499150f640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 0x7f497c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.111+0000 7f498bfff640 1 -- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 msgr2=0x7f497c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.111+0000 7f498bfff640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0701f0 0x7f497c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.111+0000 7f498bfff640 1 -- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0039b0 msgr2=0x7f497c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.111+0000 7f498bfff640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0039b0 0x7f497c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.111+0000 7f498bfff640 1 -- 172.21.15.12:0/806201925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c003520 con 0x7f497c004800 2024-04-15T15:22:17.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f498bfff640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c004800 0x7f497c004c80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f49780098a0 tx=0x7f497c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b4161deb606bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497800f040 con 0x7f497c004800 2024-04-15T15:22:17.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4978004230 con 0x7f497c004800 2024-04-15T15:22:17.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f498b7fe640 1 -- 172.21.15.12:0/806201925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978003610 con 0x7f497c004800 2024-04-15T15:22:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f4992f72640 1 -- 172.21.15.12:0/806201925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c004800 msgr2=0x7f497c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f4992f72640 1 --2- 172.21.15.12:0/806201925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f497c004800 0x7f497c004c80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f49780098a0 tx=0x7f497c005830 comp rx=0 tx=0).stop 2024-04-15T15:22:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f499150f640 1 -- 172.21.15.12:0/806201925 reap_dead start 2024-04-15T15:22:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f4992f72640 1 -- 172.21.15.12:0/806201925 shutdown_connections 2024-04-15T15:22:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f4992f72640 1 -- 172.21.15.12:0/806201925 >> 172.21.15.12:0/806201925 conn(0x7f498c0d2460 msgr2=0x7f498c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f4992f72640 1 -- 172.21.15.12:0/806201925 shutdown_connections 2024-04-15T15:22:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.112+0000 7f4992f72640 1 -- 172.21.15.12:0/806201925 wait complete. 2024-04-15T15:22:17.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.113+0000 7f4992f72640 1 Processor -- start 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.114+0000 7f4992f72640 1 -- start start 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.114+0000 7f4992f72640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c004800 0x7f498c159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.114+0000 7f4992f72640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f498c0b0d80 0x7f498c0cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.114+0000 7f4992f72640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0b1200 0x7f498c0ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.114+0000 7f4992f72640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f498c071050 con 0x7f498c0b0d80 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.114+0000 7f4992f72640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f498c0d08c0 con 0x7f497c004800 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.114+0000 7f4992f72640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f498c0b1c60 con 0x7f498c0b1200 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.114+0000 7f498bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f498c0b0d80 0x7f498c0cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f498bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f498c0b0d80 0x7f498c0cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55688/0 (socket says 172.21.15.12:55688) 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f498bfff640 1 -- 172.21.15.12:0/1632622482 learned_addr learned my addr 172.21.15.12:0/1632622482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f4990d0e640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c004800 0x7f498c159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f499150f640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0b1200 0x7f498c0ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f498bfff640 1 -- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0b1200 msgr2=0x7f498c0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f498bfff640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0b1200 0x7f498c0ce020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f498bfff640 1 -- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c004800 msgr2=0x7f498c159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f498bfff640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c004800 0x7f498c159d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f498bfff640 1 -- 172.21.15.12:0/1632622482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498c15a2d0 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f498bfff640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f498c0b0d80 0x7f498c0cdae0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f49780099c0 tx=0x7f4978004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f49897fa640 1 -- 172.21.15.12:0/1632622482 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978048070 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f49897fa640 1 -- 172.21.15.12:0/1632622482 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4978043400 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498c0af380 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f49897fa640 1 -- 172.21.15.12:0/1632622482 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49780437b0 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.115+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498c0af940 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.116+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f498c0689e0 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.117+0000 7f49897fa640 1 -- 172.21.15.12:0/1632622482 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f497804b070 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.117+0000 7f49897fa640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f497006c8f0 0x7f497006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.117+0000 7f49897fa640 1 -- 172.21.15.12:0/1632622482 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f49780ad630 con 0x7f498c0b0d80 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.117+0000 7f4990d0e640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f497006c8f0 0x7f497006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.118+0000 7f4990d0e640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f497006c8f0 0x7f497006edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4980009a60 tx=0x7f4980008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.122+0000 7f49897fa640 1 -- 172.21.15.12:0/1632622482 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4978034020 con 0x7f498c0b0d80 2024-04-15T15:22:17.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.305+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f498c0d3370 con 0x7f498c0b0d80 2024-04-15T15:22:17.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.306+0000 7f49897fa640 1 -- 172.21.15.12:0/1632622482 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f497807b060 con 0x7f498c0b0d80 2024-04-15T15:22:17.307 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:17.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f497006c8f0 msgr2=0x7f497006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f497006c8f0 0x7f497006edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4980009a60 tx=0x7f4980008040 comp rx=0 tx=0).stop 2024-04-15T15:22:17.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f498c0b0d80 msgr2=0x7f498c0cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f498c0b0d80 0x7f498c0cdae0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f49780099c0 tx=0x7f4978004730 comp rx=0 tx=0).stop 2024-04-15T15:22:17.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 shutdown_connections 2024-04-15T15:22:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f498c0b1200 0x7f498c0ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f498c0b0d80 0x7f498c0cdae0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f497006c8f0 0x7f497006edb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 --2- 172.21.15.12:0/1632622482 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c004800 0x7f498c159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 >> 172.21.15.12:0/1632622482 conn(0x7f498c0d2460 msgr2=0x7f498c1604b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 shutdown_connections 2024-04-15T15:22:17.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.308+0000 7f4992f72640 1 -- 172.21.15.12:0/1632622482 wait complete. 2024-04-15T15:22:17.320 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:17.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.424+0000 7f7fd8550640 1 Processor -- start 2024-04-15T15:22:17.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.424+0000 7f7fd8550640 1 -- start start 2024-04-15T15:22:17.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.425+0000 7f7fd8550640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7fd0070940 0x7f7fd009b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.425+0000 7f7fd8550640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fd00d05e0 con 0x7f7fd0160310 2024-04-15T15:22:17.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.425+0000 7f7fd8550640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fd015f720 con 0x7f7fd00d0da0 2024-04-15T15:22:17.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.425+0000 7f7fd8550640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd015fa00 con 0x7f7fd0070940 2024-04-15T15:22:17.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.425+0000 7f7fd5aeb640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7fd0160310 0x7f7fd0070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50924/0 (socket says 172.21.15.12:50924) 2024-04-15T15:22:17.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.425+0000 7f7fd5aeb640 1 -- 172.21.15.12:0/3843358272 learned_addr learned my addr 172.21.15.12:0/3843358272 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:17.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.425+0000 7f7fd62ec640 1 --2- 172.21.15.12:0/3843358272 >> v2:172.21.15.55:3300/0 conn(0x7f7fd0070940 0x7f7fd009b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.426+0000 7f7fd52ea640 1 -- 172.21.15.12:0/3843358272 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2307320457 0 0) 0x7f7fd00d05e0 con 0x7f7fd0160310 2024-04-15T15:22:17.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.426+0000 7f7fd52ea640 1 -- 172.21.15.12:0/3843358272 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fb8003520 con 0x7f7fd0160310 2024-04-15T15:22:17.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.426+0000 7f7fd52ea640 1 -- 172.21.15.12:0/3843358272 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699856400 0 0) 0x7f7fd015f720 con 0x7f7fd00d0da0 2024-04-15T15:22:17.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.426+0000 7f7fd52ea640 1 -- 172.21.15.12:0/3843358272 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fd00d05e0 con 0x7f7fd00d0da0 2024-04-15T15:22:17.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.426+0000 7f7fd52ea640 1 -- 172.21.15.12:0/3843358272 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 211012696 0 0) 0x7f7fb8003520 con 0x7f7fd0160310 2024-04-15T15:22:17.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.426+0000 7f7fd62ec640 1 -- 172.21.15.12:0/3843358272 >> v1:172.21.15.55:6789/0 conn(0x7f7fd00d0da0 legacy=0x7f7fd00d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.426+0000 7f7fd62ec640 1 -- 172.21.15.12:0/3843358272 >> v1:172.21.15.12:6789/0 conn(0x7f7fd0160310 legacy=0x7f7fd0070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.426+0000 7f7fd62ec640 1 -- 172.21.15.12:0/3843358272 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd015fce0 con 0x7f7fd0070940 2024-04-15T15:22:17.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd62ec640 1 --2- 172.21.15.12:0/3843358272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd0070940 0x7f7fd009b880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009d90 tx=0x7f7fc002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2822a3eb54409b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd52ea640 1 -- 172.21.15.12:0/3843358272 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0039070 con 0x7f7fd0070940 2024-04-15T15:22:17.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd52ea640 1 -- 172.21.15.12:0/3843358272 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc002c940 con 0x7f7fd0070940 2024-04-15T15:22:17.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd52ea640 1 -- 172.21.15.12:0/3843358272 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4001670 con 0x7f7fd0070940 2024-04-15T15:22:17.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd8550640 1 -- 172.21.15.12:0/3843358272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd0070940 msgr2=0x7f7fd009b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd8550640 1 --2- 172.21.15.12:0/3843358272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd0070940 0x7f7fd009b880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009d90 tx=0x7f7fc002be10 comp rx=0 tx=0).stop 2024-04-15T15:22:17.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd8550640 1 -- 172.21.15.12:0/3843358272 shutdown_connections 2024-04-15T15:22:17.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd8550640 1 --2- 172.21.15.12:0/3843358272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd0070940 0x7f7fd009b880 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd8550640 1 -- 172.21.15.12:0/3843358272 >> 172.21.15.12:0/3843358272 conn(0x7f7fd00b3f40 msgr2=0x7f7fd00b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:17.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.427+0000 7f7fd8550640 1 -- 172.21.15.12:0/3843358272 shutdown_connections 2024-04-15T15:22:17.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.428+0000 7f7fd8550640 1 -- 172.21.15.12:0/3843358272 wait complete. 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.429+0000 7f7fd8550640 1 Processor -- start 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.429+0000 7f7fd8550640 1 -- start start 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.429+0000 7f7fd8550640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7fd0070940 0x7f7fd00b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.429+0000 7f7fd8550640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00d0da0 0x7f7fd009d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.429+0000 7f7fd8550640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0160310 0x7f7fd009d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.429+0000 7f7fd8550640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd0160770 con 0x7f7fd0160310 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.429+0000 7f7fd8550640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd00d1900 con 0x7f7fd00d0da0 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.429+0000 7f7fd8550640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd015f0d0 con 0x7f7fd0070940 2024-04-15T15:22:17.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0160310 0x7f7fd009d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0160310 0x7f7fd009d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55690/0 (socket says 172.21.15.12:55690) 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 -- 172.21.15.12:0/2921322371 learned_addr learned my addr 172.21.15.12:0/2921322371 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd5aeb640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00d0da0 0x7f7fd009d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd62ec640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7fd0070940 0x7f7fd00b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 -- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7fd0070940 msgr2=0x7f7fd00b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7fd0070940 0x7f7fd00b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 -- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00d0da0 msgr2=0x7f7fd009d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00d0da0 0x7f7fd009d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 -- 172.21.15.12:0/2921322371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd009f2e0 con 0x7f7fd0160310 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd6aed640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0160310 0x7f7fd009d690 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800c400 tx=0x7f7fc800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fceffd640 1 -- 172.21.15.12:0/2921322371 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8017070 con 0x7f7fd0160310 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fceffd640 1 -- 172.21.15.12:0/2921322371 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc80052d0 con 0x7f7fd0160310 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fceffd640 1 -- 172.21.15.12:0/2921322371 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc80037f0 con 0x7f7fd0160310 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd009f5c0 con 0x7f7fd0160310 2024-04-15T15:22:17.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.430+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd009fe60 con 0x7f7fd0160310 2024-04-15T15:22:17.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.431+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd00b5040 con 0x7f7fd0160310 2024-04-15T15:22:17.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.432+0000 7f7fceffd640 1 -- 172.21.15.12:0/2921322371 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fc801a070 con 0x7f7fd0160310 2024-04-15T15:22:17.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.432+0000 7f7fceffd640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7fbc06c8f0 0x7f7fbc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.432+0000 7f7fceffd640 1 -- 172.21.15.12:0/2921322371 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f7fc808b2e0 con 0x7f7fd0160310 2024-04-15T15:22:17.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.432+0000 7f7fd62ec640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7fbc06c8f0 0x7f7fbc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.433+0000 7f7fd62ec640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7fbc06c8f0 0x7f7fbc06edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0036000 tx=0x7f7fc0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.437+0000 7f7fceffd640 1 -- 172.21.15.12:0/2921322371 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fd00b5040 con 0x7f7fd0160310 2024-04-15T15:22:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.634+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7fd00689e0 con 0x7f7fd0160310 2024-04-15T15:22:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.635+0000 7f7fceffd640 1 -- 172.21.15.12:0/2921322371 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f7fc8058bf0 con 0x7f7fd0160310 2024-04-15T15:22:17.636 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.636+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7fbc06c8f0 msgr2=0x7f7fbc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.636+0000 7f7fd8550640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7fbc06c8f0 0x7f7fbc06edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0036000 tx=0x7f7fc0030000 comp rx=0 tx=0).stop 2024-04-15T15:22:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.636+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0160310 msgr2=0x7f7fd009d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.636+0000 7f7fd8550640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0160310 0x7f7fd009d690 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800c400 tx=0x7f7fc800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.637+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 shutdown_connections 2024-04-15T15:22:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.637+0000 7f7fd8550640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0160310 0x7f7fd009d690 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.637+0000 7f7fd8550640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7fbc06c8f0 0x7f7fbc06edb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.637+0000 7f7fd8550640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00d0da0 0x7f7fd009d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.637+0000 7f7fd8550640 1 --2- 172.21.15.12:0/2921322371 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7fd0070940 0x7f7fd00b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.637+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 >> 172.21.15.12:0/2921322371 conn(0x7f7fd00b3f40 msgr2=0x7f7fd006caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.637+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 shutdown_connections 2024-04-15T15:22:17.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.637+0000 7f7fd8550640 1 -- 172.21.15.12:0/2921322371 wait complete. 2024-04-15T15:22:17.649 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi055.front.sepia.ceph.com... 2024-04-15T15:22:17.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:17.649 DEBUG:teuthology.orchestra.run.smithi055:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-15T15:22:17.683 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:17.683 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:22:17.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:17.683 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:22:17.741 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-15T15:22:17.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:22:17.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.845+0000 7f286ab52640 1 Processor -- start 2024-04-15T15:22:17.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.846+0000 7f286ab52640 1 -- start start 2024-04-15T15:22:17.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.846+0000 7f286ab52640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640d2360 0x7f28640d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.846+0000 7f286ab52640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28640b47a0 con 0x7f286415e700 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.846+0000 7f286ab52640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28640b4a80 con 0x7f286415eae0 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.846+0000 7f286ab52640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f28640b53b0 con 0x7f28640d2360 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.846+0000 7f28688ee640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f286415e700 0x7f28640b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50930/0 (socket says 172.21.15.12:50930) 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.846+0000 7f28688ee640 1 -- 172.21.15.12:0/802458344 learned_addr learned my addr 172.21.15.12:0/802458344 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f2863fff640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640d2360 0x7f28640d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 153146211 0 0) 0x7f28640b47a0 con 0x7f286415e700 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2854003520 con 0x7f286415e700 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371097188 0 0) 0x7f28640b4a80 con 0x7f286415eae0 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28640b47a0 con 0x7f286415eae0 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3859587474 0 0) 0x7f2854003520 con 0x7f286415e700 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640d2360 msgr2=0x7f28640d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640d2360 0x7f28640d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 >> v1:172.21.15.55:6789/0 conn(0x7f286415eae0 legacy=0x7f28640cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28640b4d60 con 0x7f286415e700 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f28580017f0 con 0x7f286415e700 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 >> v1:172.21.15.12:6789/0 conn(0x7f286415e700 legacy=0x7f28640b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854003b70 0x7f2854003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640d2360 0x7f2854004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2854004d40 0x7f28540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f28640b53b0 con 0x7f2854004d40 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f285400baa0 con 0x7f2854003b70 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f285400bca0 con 0x7f28640d2360 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28690ef640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2854004d40 0x7f28540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28688ee640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854003b70 0x7f2854003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f2863fff640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640d2360 0x7f2854004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28690ef640 1 -- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640d2360 msgr2=0x7f2854004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:17.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28690ef640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640d2360 0x7f2854004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28690ef640 1 -- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854003b70 msgr2=0x7f2854003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28690ef640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854003b70 0x7f2854003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.847+0000 7f28690ef640 1 -- 172.21.15.12:0/802458344 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2854003520 con 0x7f2854004d40 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.848+0000 7f28690ef640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2854004d40 0x7f28540051c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f285c001fa0 tx=0x7f2854005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f5b1bd9afcb8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.848+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c001bf0 con 0x7f2854004d40 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.848+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285c003b70 con 0x7f2854004d40 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.848+0000 7f28637fe640 1 -- 172.21.15.12:0/802458344 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c0050d0 con 0x7f2854004d40 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.849+0000 7f286ab52640 1 -- 172.21.15.12:0/802458344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2854004d40 msgr2=0x7f28540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.849+0000 7f286ab52640 1 --2- 172.21.15.12:0/802458344 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2854004d40 0x7f28540051c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f285c001fa0 tx=0x7f2854005830 comp rx=0 tx=0).stop 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.849+0000 7f28690ef640 1 -- 172.21.15.12:0/802458344 reap_dead start 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.849+0000 7f286ab52640 1 -- 172.21.15.12:0/802458344 shutdown_connections 2024-04-15T15:22:17.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.849+0000 7f286ab52640 1 -- 172.21.15.12:0/802458344 >> 172.21.15.12:0/802458344 conn(0x7f286408bf60 msgr2=0x7f28640d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.849+0000 7f286ab52640 1 -- 172.21.15.12:0/802458344 shutdown_connections 2024-04-15T15:22:17.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.849+0000 7f286ab52640 1 -- 172.21.15.12:0/802458344 wait complete. 2024-04-15T15:22:17.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.850+0000 7f286ab52640 1 Processor -- start 2024-04-15T15:22:17.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.850+0000 7f286ab52640 1 -- start start 2024-04-15T15:22:17.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f286ab52640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854004d40 0x7f28641f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f286ab52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28640c6ee0 0x7f2864160e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f286ab52640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640c9a20 0x7f28640c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f286ab52640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f28640d1d40 con 0x7f28640c6ee0 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f286ab52640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f28640b4450 con 0x7f2854004d40 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f286ab52640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f28640d1ec0 con 0x7f28640c9a20 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f2863fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28640c6ee0 0x7f2864160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f2863fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28640c6ee0 0x7f2864160e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55718/0 (socket says 172.21.15.12:55718) 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f2863fff640 1 -- 172.21.15.12:0/118116262 learned_addr learned my addr 172.21.15.12:0/118116262 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f28688ee640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854004d40 0x7f28641f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f2863fff640 1 -- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640c9a20 msgr2=0x7f28640c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f2863fff640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640c9a20 0x7f28640c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f2863fff640 1 -- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854004d40 msgr2=0x7f28641f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f2863fff640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854004d40 0x7f28641f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.851+0000 7f2863fff640 1 -- 172.21.15.12:0/118116262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28641f47a0 con 0x7f28640c6ee0 2024-04-15T15:22:17.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.852+0000 7f2863fff640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28640c6ee0 0x7f2864160e20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2850011920 tx=0x7f2850011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.852+0000 7f28617fa640 1 -- 172.21.15.12:0/118116262 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285000fa90 con 0x7f28640c6ee0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.852+0000 7f28617fa640 1 -- 172.21.15.12:0/118116262 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285000fc30 con 0x7f28640c6ee0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.852+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28640c8980 con 0x7f28640c6ee0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.852+0000 7f28617fa640 1 -- 172.21.15.12:0/118116262 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850014440 con 0x7f28640c6ee0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.852+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28640c8f40 con 0x7f28640c6ee0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.853+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28640689e0 con 0x7f28640c6ee0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.853+0000 7f28617fa640 1 -- 172.21.15.12:0/118116262 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2850012040 con 0x7f28640c6ee0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.854+0000 7f28617fa640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f284806c8f0 0x7f284806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.854+0000 7f28617fa640 1 -- 172.21.15.12:0/118116262 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f285008c650 con 0x7f28640c6ee0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.854+0000 7f28688ee640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f284806c8f0 0x7f284806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:17.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.854+0000 7f28688ee640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f284806c8f0 0x7f284806edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f28580011d0 tx=0x7f2858009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:17.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:17.858+0000 7f28617fa640 1 -- 172.21.15.12:0/118116262 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2850019030 con 0x7f28640c6ee0 2024-04-15T15:22:18.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.082+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28640ba500 con 0x7f28640c6ee0 2024-04-15T15:22:18.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.083+0000 7f28617fa640 1 -- 172.21.15.12:0/118116262 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f285005a5b0 con 0x7f28640c6ee0 2024-04-15T15:22:18.085 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:18.085 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:33.113220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:22:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f284806c8f0 msgr2=0x7f284806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f284806c8f0 0x7f284806edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f28580011d0 tx=0x7f2858009040 comp rx=0 tx=0).stop 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28640c6ee0 msgr2=0x7f2864160e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28640c6ee0 0x7f2864160e20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2850011920 tx=0x7f2850011de0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 shutdown_connections 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28640c9a20 0x7f28640c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28640c6ee0 0x7f2864160e20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f284806c8f0 0x7f284806edb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 --2- 172.21.15.12:0/118116262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2854004d40 0x7f28641f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 >> 172.21.15.12:0/118116262 conn(0x7f286408bf60 msgr2=0x7f28640d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 shutdown_connections 2024-04-15T15:22:18.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.085+0000 7f286ab52640 1 -- 172.21.15.12:0/118116262 wait complete. 2024-04-15T15:22:18.098 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.201+0000 7f624b700640 1 Processor -- start 2024-04-15T15:22:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.201+0000 7f624b700640 1 -- start start 2024-04-15T15:22:18.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f624b700640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f624415e700 0x7f62440b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f624b700640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 0x7f62440cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f624b700640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62440b47a0 con 0x7f62440d2360 2024-04-15T15:22:18.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f624b700640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f62440b53b0 con 0x7f624415e700 2024-04-15T15:22:18.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f624b700640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f62440b4a80 con 0x7f62440d3270 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f6248c9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f62440d2360 0x7f62440d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50942/0 (socket says 172.21.15.12:50942) 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f6248c9b640 1 -- 172.21.15.12:0/1723325682 learned_addr learned my addr 172.21.15.12:0/1723325682 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f624949c640 1 --2- 172.21.15.12:0/1723325682 >> v2:172.21.15.55:3300/0 conn(0x7f624415e700 0x7f62440b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988585251 0 0) 0x7f62440b47a0 con 0x7f62440d2360 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f6249c9d640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 0x7f62440cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6230003520 con 0x7f62440d2360 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4073026302 0 0) 0x7f6230003520 con 0x7f62440d2360 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 msgr2=0x7f62440cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 0x7f62440cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 >> v2:172.21.15.55:3300/0 conn(0x7f624415e700 msgr2=0x7f62440b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 --2- 172.21.15.12:0/1723325682 >> v2:172.21.15.55:3300/0 conn(0x7f624415e700 0x7f62440b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62440b4d60 con 0x7f62440d2360 2024-04-15T15:22:18.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f6234001670 con 0x7f62440d2360 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 >> v1:172.21.15.12:6789/0 conn(0x7f62440d2360 legacy=0x7f62440d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300039b0 0x7f6230003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 0x7f6230004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6230004d40 0x7f62300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f62440b53b0 con 0x7f62300039b0 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f62440b4a80 con 0x7f6230004d40 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.202+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f623000b9c0 con 0x7f62440d3270 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f624949c640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300039b0 0x7f6230003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f6248c9b640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 0x7f6230004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f6249c9d640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6230004d40 0x7f62300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f624949c640 1 -- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 msgr2=0x7f6230004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f624949c640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 0x7f6230004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f624949c640 1 -- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6230004d40 msgr2=0x7f62300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:18.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f624949c640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6230004d40 0x7f62300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f624949c640 1 -- 172.21.15.12:0/1723325682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6230003520 con 0x7f62300039b0 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f6248c9b640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440d3270 0x7f6230004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.203+0000 7f624949c640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300039b0 0x7f6230003dd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6240009ae0 tx=0x7f6230005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b39ea91e1ea769 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240009680 con 0x7f62300039b0 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6240005990 con 0x7f62300039b0 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f623bfff640 1 -- 172.21.15.12:0/1723325682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240005d80 con 0x7f62300039b0 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f624b700640 1 -- 172.21.15.12:0/1723325682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300039b0 msgr2=0x7f6230003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f624b700640 1 --2- 172.21.15.12:0/1723325682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300039b0 0x7f6230003dd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6240009ae0 tx=0x7f6230005830 comp rx=0 tx=0).stop 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f6249c9d640 1 -- 172.21.15.12:0/1723325682 reap_dead start 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f624b700640 1 -- 172.21.15.12:0/1723325682 shutdown_connections 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f624b700640 1 -- 172.21.15.12:0/1723325682 >> 172.21.15.12:0/1723325682 conn(0x7f624408bf60 msgr2=0x7f62440d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f624b700640 1 -- 172.21.15.12:0/1723325682 shutdown_connections 2024-04-15T15:22:18.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.204+0000 7f624b700640 1 -- 172.21.15.12:0/1723325682 wait complete. 2024-04-15T15:22:18.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.205+0000 7f624b700640 1 Processor -- start 2024-04-15T15:22:18.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f624b700640 1 -- start start 2024-04-15T15:22:18.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f624b700640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62300039b0 0x7f624415cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f624b700640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62440b8600 0x7f62440b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f624b700640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440b26d0 0x7f62440b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f624b700640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f62440b5d80 con 0x7f62440b8600 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f624b700640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f62440b4450 con 0x7f62300039b0 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f624b700640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f62440d1d40 con 0x7f62440b26d0 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f6248c9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62440b8600 0x7f62440b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f6248c9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62440b8600 0x7f62440b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55738/0 (socket says 172.21.15.12:55738) 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.206+0000 7f6248c9b640 1 -- 172.21.15.12:0/1944220289 learned_addr learned my addr 172.21.15.12:0/1944220289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f624949c640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62300039b0 0x7f624415cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6248c9b640 1 -- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440b26d0 msgr2=0x7f62440b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6248c9b640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440b26d0 0x7f62440b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6248c9b640 1 -- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62300039b0 msgr2=0x7f624415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6248c9b640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62300039b0 0x7f624415cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6248c9b640 1 -- 172.21.15.12:0/1944220289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62440b7560 con 0x7f62440b8600 2024-04-15T15:22:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6248c9b640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62440b8600 0x7f62440b2190 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6234000fb0 tx=0x7f623400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6239ffb640 1 -- 172.21.15.12:0/1944220289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623400ee60 con 0x7f62440b8600 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6239ffb640 1 -- 172.21.15.12:0/1944220289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6234018070 con 0x7f62440b8600 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f6239ffb640 1 -- 172.21.15.12:0/1944220289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623400ca70 con 0x7f62440b8600 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62440b7840 con 0x7f62440b8600 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.207+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62440b7e00 con 0x7f62440b8600 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.208+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62440689e0 con 0x7f62440b8600 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.208+0000 7f6239ffb640 1 -- 172.21.15.12:0/1944220289 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f623400cc10 con 0x7f62440b8600 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.209+0000 7f6239ffb640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f622806c8f0 0x7f622806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.209+0000 7f6239ffb640 1 -- 172.21.15.12:0/1944220289 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f623408bdc0 con 0x7f62440b8600 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.209+0000 7f624949c640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f622806c8f0 0x7f622806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.210+0000 7f624949c640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f622806c8f0 0x7f622806edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6240035000 tx=0x7f6240030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.214+0000 7f6239ffb640 1 -- 172.21.15.12:0/1944220289 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6234059db0 con 0x7f62440b8600 2024-04-15T15:22:18.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.399+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62440ba500 con 0x7f62440b8600 2024-04-15T15:22:18.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.400+0000 7f6239ffb640 1 -- 172.21.15.12:0/1944220289 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f62340596b0 con 0x7f62440b8600 2024-04-15T15:22:18.402 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:18.402 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f622806c8f0 msgr2=0x7f622806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f622806c8f0 0x7f622806edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6240035000 tx=0x7f6240030000 comp rx=0 tx=0).stop 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62440b8600 msgr2=0x7f62440b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62440b8600 0x7f62440b2190 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6234000fb0 tx=0x7f623400ec10 comp rx=0 tx=0).stop 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 shutdown_connections 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62440b26d0 0x7f62440b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62440b8600 0x7f62440b2190 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f622806c8f0 0x7f622806edb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 --2- 172.21.15.12:0/1944220289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62300039b0 0x7f624415cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 >> 172.21.15.12:0/1944220289 conn(0x7f624408bf60 msgr2=0x7f624415eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 shutdown_connections 2024-04-15T15:22:18.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.402+0000 7f624b700640 1 -- 172.21.15.12:0/1944220289 wait complete. 2024-04-15T15:22:18.415 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:21:08.716989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:21:09.952110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:21:13.669291+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:22:18.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.521+0000 7faa79782640 1 Processor -- start 2024-04-15T15:22:18.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.521+0000 7faa79782640 1 -- start start 2024-04-15T15:22:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa79782640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa74160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa79782640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa7406f090 con 0x7faa74067720 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa79782640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa7415d660 con 0x7faa7415d090 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa79782640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faa74160130 con 0x7faa7406ec00 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa727fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7faa74067720 0x7faa740ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50950/0 (socket says 172.21.15.12:50950) 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa727fc640 1 -- 172.21.15.12:0/2217370549 learned_addr learned my addr 172.21.15.12:0/2217370549 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa737fe640 1 --2- 172.21.15.12:0/2217370549 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa74160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa71ffb640 1 -- 172.21.15.12:0/2217370549 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255652294 0 0) 0x7faa7406f090 con 0x7faa74067720 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa71ffb640 1 -- 172.21.15.12:0/2217370549 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa5c003520 con 0x7faa74067720 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa71ffb640 1 -- 172.21.15.12:0/2217370549 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 898477520 0 0) 0x7faa7415d660 con 0x7faa7415d090 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa71ffb640 1 -- 172.21.15.12:0/2217370549 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa7406f090 con 0x7faa7415d090 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa737fe640 1 -- 172.21.15.12:0/2217370549 >> v1:172.21.15.55:6789/0 conn(0x7faa7415d090 legacy=0x7faa7406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa71ffb640 1 -- 172.21.15.12:0/2217370549 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 750459976 0 0) 0x7faa5c003520 con 0x7faa74067720 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa737fe640 1 -- 172.21.15.12:0/2217370549 >> v1:172.21.15.12:6789/0 conn(0x7faa74067720 legacy=0x7faa740ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.522+0000 7faa737fe640 1 -- 172.21.15.12:0/2217370549 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7415d940 con 0x7faa7406ec00 2024-04-15T15:22:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.523+0000 7faa737fe640 1 --2- 172.21.15.12:0/2217370549 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa74160b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009dc0 tx=0x7faa6c02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=713fff257c11adc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.523+0000 7faa71ffb640 1 -- 172.21.15.12:0/2217370549 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c039070 con 0x7faa7406ec00 2024-04-15T15:22:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.523+0000 7faa71ffb640 1 -- 172.21.15.12:0/2217370549 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa6c02c880 con 0x7faa7406ec00 2024-04-15T15:22:18.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.523+0000 7faa71ffb640 1 -- 172.21.15.12:0/2217370549 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa68001670 con 0x7faa7406ec00 2024-04-15T15:22:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.524+0000 7faa79782640 1 -- 172.21.15.12:0/2217370549 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 msgr2=0x7faa74160b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.524+0000 7faa79782640 1 --2- 172.21.15.12:0/2217370549 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa74160b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009dc0 tx=0x7faa6c02bd50 comp rx=0 tx=0).stop 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.524+0000 7faa79782640 1 -- 172.21.15.12:0/2217370549 shutdown_connections 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.524+0000 7faa79782640 1 --2- 172.21.15.12:0/2217370549 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa74160b80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.524+0000 7faa79782640 1 -- 172.21.15.12:0/2217370549 >> 172.21.15.12:0/2217370549 conn(0x7faa74073360 msgr2=0x7faa7406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.524+0000 7faa79782640 1 -- 172.21.15.12:0/2217370549 shutdown_connections 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.524+0000 7faa79782640 1 -- 172.21.15.12:0/2217370549 wait complete. 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.525+0000 7faa79782640 1 Processor -- start 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.525+0000 7faa79782640 1 -- start start 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa79782640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa74067720 0x7faa74155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa79782640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa740b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa79782640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa7415d090 0x7faa740b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa79782640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa74160660 con 0x7faa7415d090 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa79782640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faa741610c0 con 0x7faa74067720 2024-04-15T15:22:18.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa79782640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faa740b0460 con 0x7faa7406ec00 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa7415d090 0x7faa740b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa7415d090 0x7faa740b4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55752/0 (socket says 172.21.15.12:55752) 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa737fe640 1 -- 172.21.15.12:0/1651077520 learned_addr learned my addr 172.21.15.12:0/1651077520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa72ffd640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa74067720 0x7faa74155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa727fc640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa740b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa737fe640 1 -- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 msgr2=0x7faa740b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa737fe640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa740b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa737fe640 1 -- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa74067720 msgr2=0x7faa74155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa737fe640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa74067720 0x7faa74155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.526+0000 7faa737fe640 1 -- 172.21.15.12:0/1651077520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa740c5db0 con 0x7faa7415d090 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.527+0000 7faa737fe640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa7415d090 0x7faa740b4ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009ee0 tx=0x7faa6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.527+0000 7faa5b7fe640 1 -- 172.21.15.12:0/1651077520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c039050 con 0x7faa7415d090 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.527+0000 7faa5b7fe640 1 -- 172.21.15.12:0/1651077520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa6c045070 con 0x7faa7415d090 2024-04-15T15:22:18.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.527+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa740b5950 con 0x7faa7415d090 2024-04-15T15:22:18.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.527+0000 7faa5b7fe640 1 -- 172.21.15.12:0/1651077520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c009450 con 0x7faa7415d090 2024-04-15T15:22:18.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.527+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa740ada90 con 0x7faa7415d090 2024-04-15T15:22:18.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.528+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa740b6e60 con 0x7faa7415d090 2024-04-15T15:22:18.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.528+0000 7faa5b7fe640 1 -- 172.21.15.12:0/1651077520 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faa6c040690 con 0x7faa7415d090 2024-04-15T15:22:18.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.529+0000 7faa5b7fe640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa6006c8f0 0x7faa6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.529+0000 7faa5b7fe640 1 -- 172.21.15.12:0/1651077520 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7faa6c0ae570 con 0x7faa7415d090 2024-04-15T15:22:18.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.529+0000 7faa72ffd640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa6006c8f0 0x7faa6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.530+0000 7faa72ffd640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa6006c8f0 0x7faa6006edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faa640011e0 tx=0x7faa64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.534+0000 7faa5b7fe640 1 -- 172.21.15.12:0/1651077520 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa740b6e60 con 0x7faa7415d090 2024-04-15T15:22:18.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.719+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa740689e0 con 0x7faa7415d090 2024-04-15T15:22:18.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.719+0000 7faa5b7fe640 1 -- 172.21.15.12:0/1651077520 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7faa6c07bfa0 con 0x7faa7415d090 2024-04-15T15:22:18.721 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa6006c8f0 msgr2=0x7faa6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa6006c8f0 0x7faa6006edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faa640011e0 tx=0x7faa64009040 comp rx=0 tx=0).stop 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa7415d090 msgr2=0x7faa740b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa7415d090 0x7faa740b4ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009ee0 tx=0x7faa6c004a40 comp rx=0 tx=0).stop 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 shutdown_connections 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa7415d090 0x7faa740b4ff0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faa7406ec00 0x7faa740b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7faa6006c8f0 0x7faa6006edb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 --2- 172.21.15.12:0/1651077520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa74067720 0x7faa74155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 >> 172.21.15.12:0/1651077520 conn(0x7faa74073360 msgr2=0x7faa74074df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:18.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 shutdown_connections 2024-04-15T15:22:18.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.721+0000 7faa79782640 1 -- 172.21.15.12:0/1651077520 wait complete. 2024-04-15T15:22:18.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:18.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.840+0000 7f5412e7b640 1 Processor -- start 2024-04-15T15:22:18.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.840+0000 7f5412e7b640 1 -- start start 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5412e7b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f540c0d3240 0x7f540c0d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5412e7b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f540c0d1440 0x7f540c0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5412e7b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540c0d2480 con 0x7f540c15e790 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5412e7b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f540c0d2db0 con 0x7f540c0d3240 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5412e7b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f540c0d2760 con 0x7f540c0d1440 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5403fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f540c0d3240 0x7f540c0d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5410c17640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f540c15e790 0x7f540c0cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50958/0 (socket says 172.21.15.12:50958) 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5410c17640 1 -- 172.21.15.12:0/3080147762 learned_addr learned my addr 172.21.15.12:0/3080147762 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5403fff640 1 --2- 172.21.15.12:0/3080147762 >> v2:172.21.15.12:3300/0 conn(0x7f540c0d3240 0x7f540c0d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55756/0 (socket says 172.21.15.12:55756) 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5411418640 1 --2- 172.21.15.12:0/3080147762 >> v2:172.21.15.55:3300/0 conn(0x7f540c0d1440 0x7f540c0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5403fff640 1 -- 172.21.15.12:0/3080147762 >> v2:172.21.15.55:3300/0 conn(0x7f540c0d1440 msgr2=0x7f540c0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:18.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5403fff640 1 --2- 172.21.15.12:0/3080147762 >> v2:172.21.15.55:3300/0 conn(0x7f540c0d1440 0x7f540c0b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5403fff640 1 -- 172.21.15.12:0/3080147762 >> v1:172.21.15.12:6789/0 conn(0x7f540c15e790 legacy=0x7f540c0cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.841+0000 7f5403fff640 1 -- 172.21.15.12:0/3080147762 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540c0d2a40 con 0x7f540c0d3240 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.842+0000 7f5403fff640 1 --2- 172.21.15.12:0/3080147762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d3240 0x7f540c0d0f00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f540c0c8120 tx=0x7f53fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2c4f45fd628864 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.842+0000 7f54037fe640 1 -- 172.21.15.12:0/3080147762 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc035020 con 0x7f540c0d3240 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.842+0000 7f54037fe640 1 -- 172.21.15.12:0/3080147762 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53fc031830 con 0x7f540c0d3240 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.842+0000 7f54037fe640 1 -- 172.21.15.12:0/3080147762 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc031c00 con 0x7f540c0d3240 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.843+0000 7f5412e7b640 1 -- 172.21.15.12:0/3080147762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d3240 msgr2=0x7f540c0d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.843+0000 7f5412e7b640 1 --2- 172.21.15.12:0/3080147762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d3240 0x7f540c0d0f00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f540c0c8120 tx=0x7f53fc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.843+0000 7f5412e7b640 1 -- 172.21.15.12:0/3080147762 shutdown_connections 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.843+0000 7f5412e7b640 1 --2- 172.21.15.12:0/3080147762 >> v2:172.21.15.55:3300/0 conn(0x7f540c0d1440 0x7f540c0b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.843+0000 7f5412e7b640 1 --2- 172.21.15.12:0/3080147762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d3240 0x7f540c0d0f00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.843+0000 7f5412e7b640 1 -- 172.21.15.12:0/3080147762 >> 172.21.15.12:0/3080147762 conn(0x7f540c08bf60 msgr2=0x7f540c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.843+0000 7f5412e7b640 1 -- 172.21.15.12:0/3080147762 shutdown_connections 2024-04-15T15:22:18.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.843+0000 7f5412e7b640 1 -- 172.21.15.12:0/3080147762 wait complete. 2024-04-15T15:22:18.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.844+0000 7f5412e7b640 1 Processor -- start 2024-04-15T15:22:18.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.844+0000 7f5412e7b640 1 -- start start 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5412e7b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d1440 0x7f540c14c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5412e7b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f540c0d3240 0x7f540c1e8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5412e7b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f540c15e790 0x7f540c0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5412e7b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f540c0d5180 con 0x7f540c0d1440 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5412e7b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f540c0d0430 con 0x7f540c0d3240 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5412e7b640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f540c0d5300 con 0x7f540c15e790 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d1440 0x7f540c14c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d1440 0x7f540c14c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55766/0 (socket says 172.21.15.12:55766) 2024-04-15T15:22:18.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 -- 172.21.15.12:0/1247019263 learned_addr learned my addr 172.21.15.12:0/1247019263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5403fff640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f540c0d3240 0x7f540c1e8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5411418640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f540c15e790 0x7f540c0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 -- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f540c15e790 msgr2=0x7f540c0b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f540c15e790 0x7f540c0b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 -- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f540c0d3240 msgr2=0x7f540c1e8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f540c0d3240 0x7f540c1e8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 -- 172.21.15.12:0/1247019263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540c1e9050 con 0x7f540c0d1440 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.845+0000 7f5410c17640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d1440 0x7f540c14c070 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f540800ca70 tx=0x7f540800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.846+0000 7f54017fa640 1 -- 172.21.15.12:0/1247019263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408017070 con 0x7f540c0d1440 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.846+0000 7f54017fa640 1 -- 172.21.15.12:0/1247019263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f540800abd0 con 0x7f540c0d1440 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.846+0000 7f54017fa640 1 -- 172.21.15.12:0/1247019263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408012400 con 0x7f540c0d1440 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.846+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f540c0c9b60 con 0x7f540c0d1440 2024-04-15T15:22:18.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.846+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f540c0ca120 con 0x7f540c0d1440 2024-04-15T15:22:18.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.847+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f540c0689e0 con 0x7f540c0d1440 2024-04-15T15:22:18.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.847+0000 7f54017fa640 1 -- 172.21.15.12:0/1247019263 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f540801a070 con 0x7f540c0d1440 2024-04-15T15:22:18.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.848+0000 7f54017fa640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f53f806c8f0 0x7f53f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:18.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.848+0000 7f54017fa640 1 -- 172.21.15.12:0/1247019263 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f540808bcc0 con 0x7f540c0d1440 2024-04-15T15:22:18.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.848+0000 7f5403fff640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f53f806c8f0 0x7f53f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:18.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.848+0000 7f5403fff640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f53f806c8f0 0x7f53f806edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f53fc036000 tx=0x7f53fc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:18.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:18.852+0000 7f54017fa640 1 -- 172.21.15.12:0/1247019263 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5408092080 con 0x7f540c0d1440 2024-04-15T15:22:19.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.036+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f540c0d1ce0 con 0x7f540c0d1440 2024-04-15T15:22:19.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.036+0000 7f54017fa640 1 -- 172.21.15.12:0/1247019263 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f54080595c0 con 0x7f540c0d1440 2024-04-15T15:22:19.038 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f53f806c8f0 msgr2=0x7f53f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f53f806c8f0 0x7f53f806edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f53fc036000 tx=0x7f53fc008030 comp rx=0 tx=0).stop 2024-04-15T15:22:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d1440 msgr2=0x7f540c14c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d1440 0x7f540c14c070 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f540800ca70 tx=0x7f540800cf30 comp rx=0 tx=0).stop 2024-04-15T15:22:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 shutdown_connections 2024-04-15T15:22:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f540c15e790 0x7f540c0b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f53f806c8f0 0x7f53f806edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f540c0d3240 0x7f540c1e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 --2- 172.21.15.12:0/1247019263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f540c0d1440 0x7f540c14c070 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.038+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 >> 172.21.15.12:0/1247019263 conn(0x7f540c08bf60 msgr2=0x7f540c0b9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:19.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.039+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 shutdown_connections 2024-04-15T15:22:19.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.039+0000 7f5412e7b640 1 -- 172.21.15.12:0/1247019263 wait complete. 2024-04-15T15:22:19.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.157+0000 7f781e18d640 1 Processor -- start 2024-04-15T15:22:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.157+0000 7f781e18d640 1 -- start start 2024-04-15T15:22:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f781e18d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f781809a9d0 0x7f7818160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f781e18d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781815fb30 0x7f78180d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f781e18d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7818070720 con 0x7f78180701f0 2024-04-15T15:22:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f781e18d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f781809ba90 con 0x7f781809a9d0 2024-04-15T15:22:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f781e18d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f78180d08c0 con 0x7f781815fb30 2024-04-15T15:22:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f78177fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f781809a9d0 0x7f7818160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f7817fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781815fb30 0x7f78180d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f7816ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f78180701f0 0x7f781809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:50198/0 (socket says 172.21.15.12:50198) 2024-04-15T15:22:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.158+0000 7f7816ffd640 1 -- 172.21.15.12:0/3511791898 learned_addr learned my addr 172.21.15.12:0/3511791898 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.159+0000 7f78177fe640 1 -- 172.21.15.12:0/3511791898 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781815fb30 msgr2=0x7f78180d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.159+0000 7f78177fe640 1 --2- 172.21.15.12:0/3511791898 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781815fb30 0x7f78180d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.159+0000 7f78177fe640 1 -- 172.21.15.12:0/3511791898 >> v1:172.21.15.55:6789/0 conn(0x7f78180701f0 legacy=0x7f781809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.159+0000 7f78177fe640 1 -- 172.21.15.12:0/3511791898 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7818070a90 con 0x7f781809a9d0 2024-04-15T15:22:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.159+0000 7f7817fff640 1 --2- 172.21.15.12:0/3511791898 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781815fb30 0x7f78180d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f78177fe640 1 --2- 172.21.15.12:0/3511791898 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f781809a9d0 0x7f7818160420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7808009d90 tx=0x7f780802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e960c517fae9b44 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f78167fc640 1 -- 172.21.15.12:0/3511791898 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808039070 con 0x7f781809a9d0 2024-04-15T15:22:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f78167fc640 1 -- 172.21.15.12:0/3511791898 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f780802cc50 con 0x7f781809a9d0 2024-04-15T15:22:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f78167fc640 1 -- 172.21.15.12:0/3511791898 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808034440 con 0x7f781809a9d0 2024-04-15T15:22:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f781e18d640 1 -- 172.21.15.12:0/3511791898 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f781809a9d0 msgr2=0x7f7818160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f781e18d640 1 --2- 172.21.15.12:0/3511791898 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f781809a9d0 0x7f7818160420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7808009d90 tx=0x7f780802bea0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f781e18d640 1 -- 172.21.15.12:0/3511791898 shutdown_connections 2024-04-15T15:22:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f781e18d640 1 --2- 172.21.15.12:0/3511791898 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781815fb30 0x7f78180d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f781e18d640 1 --2- 172.21.15.12:0/3511791898 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f781809a9d0 0x7f7818160420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f781e18d640 1 -- 172.21.15.12:0/3511791898 >> 172.21.15.12:0/3511791898 conn(0x7f78180d2460 msgr2=0x7f78180d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f781e18d640 1 -- 172.21.15.12:0/3511791898 shutdown_connections 2024-04-15T15:22:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.160+0000 7f781e18d640 1 -- 172.21.15.12:0/3511791898 wait complete. 2024-04-15T15:22:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.161+0000 7f781e18d640 1 Processor -- start 2024-04-15T15:22:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.162+0000 7f781e18d640 1 -- start start 2024-04-15T15:22:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.162+0000 7f781e18d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78180701f0 0x7f78180c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.162+0000 7f781e18d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781809a9d0 0x7f781815e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.162+0000 7f781e18d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781815fb30 0x7f78181e8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.162+0000 7f781e18d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f781815ffc0 con 0x7f781815fb30 2024-04-15T15:22:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.162+0000 7f781e18d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7818071050 con 0x7f78180701f0 2024-04-15T15:22:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.162+0000 7f781e18d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f78180b1c60 con 0x7f781809a9d0 2024-04-15T15:22:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.162+0000 7f7817fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781815fb30 0x7f78181e8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7817fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781815fb30 0x7f78181e8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55768/0 (socket says 172.21.15.12:55768) 2024-04-15T15:22:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7817fff640 1 -- 172.21.15.12:0/507058330 learned_addr learned my addr 172.21.15.12:0/507058330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f78177fe640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78180701f0 0x7f78180c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7816ffd640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781809a9d0 0x7f781815e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7817fff640 1 -- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781809a9d0 msgr2=0x7f781815e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7817fff640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781809a9d0 0x7f781815e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7817fff640 1 -- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78180701f0 msgr2=0x7f78180c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7817fff640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78180701f0 0x7f78180c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7817fff640 1 -- 172.21.15.12:0/507058330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78180b6bd0 con 0x7f781815fb30 2024-04-15T15:22:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f7817fff640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781815fb30 0x7f78181e8540 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f781000e8b0 tx=0x7f781000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f77fffff640 1 -- 172.21.15.12:0/507058330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7810017070 con 0x7f781815fb30 2024-04-15T15:22:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f77fffff640 1 -- 172.21.15.12:0/507058330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7810004020 con 0x7f781815fb30 2024-04-15T15:22:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f77fffff640 1 -- 172.21.15.12:0/507058330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78100043f0 con 0x7f781815fb30 2024-04-15T15:22:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78180b7950 con 0x7f781815fb30 2024-04-15T15:22:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.163+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78180b7150 con 0x7f781815fb30 2024-04-15T15:22:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.164+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78180689e0 con 0x7f781815fb30 2024-04-15T15:22:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.165+0000 7f77fffff640 1 -- 172.21.15.12:0/507058330 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f781000c940 con 0x7f781815fb30 2024-04-15T15:22:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.165+0000 7f77fffff640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f780406c8f0 0x7f780406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.165+0000 7f77fffff640 1 -- 172.21.15.12:0/507058330 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f781008bd40 con 0x7f781815fb30 2024-04-15T15:22:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.165+0000 7f78177fe640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f780406c8f0 0x7f780406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.166+0000 7f78177fe640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f780406c8f0 0x7f780406edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7808036000 tx=0x7f7808030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.170+0000 7f77fffff640 1 -- 172.21.15.12:0/507058330 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78100596e0 con 0x7f781815fb30 2024-04-15T15:22:19.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.354+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78180b83e0 con 0x7f781815fb30 2024-04-15T15:22:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.354+0000 7f77fffff640 1 -- 172.21.15.12:0/507058330 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f781005d9e0 con 0x7f781815fb30 2024-04-15T15:22:19.356 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f780406c8f0 msgr2=0x7f780406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f780406c8f0 0x7f780406edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7808036000 tx=0x7f7808030000 comp rx=0 tx=0).stop 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781815fb30 msgr2=0x7f78181e8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781815fb30 0x7f78181e8540 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f781000e8b0 tx=0x7f781000ed70 comp rx=0 tx=0).stop 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 shutdown_connections 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781815fb30 0x7f78181e8540 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f781809a9d0 0x7f781815e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f780406c8f0 0x7f780406edb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 --2- 172.21.15.12:0/507058330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78180701f0 0x7f78180c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 >> 172.21.15.12:0/507058330 conn(0x7f78180d2460 msgr2=0x7f78181626f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:19.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 shutdown_connections 2024-04-15T15:22:19.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.356+0000 7f781e18d640 1 -- 172.21.15.12:0/507058330 wait complete. 2024-04-15T15:22:19.369 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:22:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.475+0000 7f22ef3bd640 1 Processor -- start 2024-04-15T15:22:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.475+0000 7f22ef3bd640 1 -- start start 2024-04-15T15:22:19.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ef3bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22e80d2360 0x7f22e80d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ef3bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e815eae0 0x7f22e80d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ef3bd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80b47a0 con 0x7f22e815e700 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ef3bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f22e80b53b0 con 0x7f22e80d2360 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ef3bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f22e80b4a80 con 0x7f22e815eae0 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ec958640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22e80d2360 0x7f22e80d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ec958640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22e80d2360 0x7f22e80d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55782/0 (socket says 172.21.15.12:55782) 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ec958640 1 -- 172.21.15.12:0/2273551089 learned_addr learned my addr 172.21.15.12:0/2273551089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ec958640 1 -- 172.21.15.12:0/2273551089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e815eae0 msgr2=0x7f22e80d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ec958640 1 --2- 172.21.15.12:0/2273551089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e815eae0 0x7f22e80d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ec958640 1 -- 172.21.15.12:0/2273551089 >> v1:172.21.15.55:6789/0 conn(0x7f22e815e700 legacy=0x7f22e80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.476+0000 7f22ec958640 1 -- 172.21.15.12:0/2273551089 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e80b4d60 con 0x7f22e80d2360 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.477+0000 7f22ec958640 1 --2- 172.21.15.12:0/2273551089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e80d2360 0x7f22e80d2b60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f22e808c7b0 tx=0x7f22dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e7a920b8862e57 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.477+0000 7f22e7fff640 1 -- 172.21.15.12:0/2273551089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc035020 con 0x7f22e80d2360 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.477+0000 7f22e7fff640 1 -- 172.21.15.12:0/2273551089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22dc031830 con 0x7f22e80d2360 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.477+0000 7f22e7fff640 1 -- 172.21.15.12:0/2273551089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc031c00 con 0x7f22e80d2360 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.478+0000 7f22ef3bd640 1 -- 172.21.15.12:0/2273551089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e80d2360 msgr2=0x7f22e80d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.478+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/2273551089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e80d2360 0x7f22e80d2b60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f22e808c7b0 tx=0x7f22dc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.478+0000 7f22ef3bd640 1 -- 172.21.15.12:0/2273551089 shutdown_connections 2024-04-15T15:22:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.478+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/2273551089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e80d2360 0x7f22e80d2b60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.478+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/2273551089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e815eae0 0x7f22e80d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.478+0000 7f22ef3bd640 1 -- 172.21.15.12:0/2273551089 >> 172.21.15.12:0/2273551089 conn(0x7f22e808bf60 msgr2=0x7f22e80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.478+0000 7f22ef3bd640 1 -- 172.21.15.12:0/2273551089 shutdown_connections 2024-04-15T15:22:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.478+0000 7f22ef3bd640 1 -- 172.21.15.12:0/2273551089 wait complete. 2024-04-15T15:22:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.479+0000 7f22ef3bd640 1 Processor -- start 2024-04-15T15:22:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.479+0000 7f22ef3bd640 1 -- start start 2024-04-15T15:22:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ef3bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e80d2360 0x7f22e80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ef3bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e815e700 0x7f22e8160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ef3bd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22e815eae0 0x7f22e80cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ef3bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f22e80d1d40 con 0x7f22e815e700 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ef3bd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f22e80b4450 con 0x7f22e815eae0 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ef3bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f22e80d1ec0 con 0x7f22e80d2360 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e815e700 0x7f22e8160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e815e700 0x7f22e8160030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55796/0 (socket says 172.21.15.12:55796) 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 -- 172.21.15.12:0/3858241788 learned_addr learned my addr 172.21.15.12:0/3858241788 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ed95a640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22e815eae0 0x7f22e80cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ed159640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e80d2360 0x7f22e80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 -- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e80d2360 msgr2=0x7f22e80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e80d2360 0x7f22e80ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 -- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22e815eae0 msgr2=0x7f22e80cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22e815eae0 0x7f22e80cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 -- 172.21.15.12:0/3858241788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e80cbb50 con 0x7f22e815e700 2024-04-15T15:22:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ed95a640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22e815eae0 0x7f22e80cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ed159640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e80d2360 0x7f22e80ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.480+0000 7f22ec958640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e815e700 0x7f22e8160030 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f22dc000c00 tx=0x7f22dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.481+0000 7f22e5ffb640 1 -- 172.21.15.12:0/3858241788 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc03a930 con 0x7f22e815e700 2024-04-15T15:22:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.481+0000 7f22e5ffb640 1 -- 172.21.15.12:0/3858241788 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22dc03aad0 con 0x7f22e815e700 2024-04-15T15:22:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.481+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e80cbe30 con 0x7f22e815e700 2024-04-15T15:22:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.481+0000 7f22e5ffb640 1 -- 172.21.15.12:0/3858241788 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc0443f0 con 0x7f22e815e700 2024-04-15T15:22:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.481+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e80cc3f0 con 0x7f22e815e700 2024-04-15T15:22:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.482+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e80689e0 con 0x7f22e815e700 2024-04-15T15:22:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.482+0000 7f22e5ffb640 1 -- 172.21.15.12:0/3858241788 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f22dc04b060 con 0x7f22e815e700 2024-04-15T15:22:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.483+0000 7f22e5ffb640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22d406c8f0 0x7f22d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.483+0000 7f22e5ffb640 1 -- 172.21.15.12:0/3858241788 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f22dc0add10 con 0x7f22e815e700 2024-04-15T15:22:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.483+0000 7f22ed159640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22d406c8f0 0x7f22d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.484+0000 7f22ed159640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22d406c8f0 0x7f22d406edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f22e0003e60 tx=0x7f22e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.487+0000 7f22e5ffb640 1 -- 172.21.15.12:0/3858241788 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f22dc03f070 con 0x7f22e815e700 2024-04-15T15:22:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.674+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f22e80ba500 con 0x7f22e815e700 2024-04-15T15:22:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.675+0000 7f22e5ffb640 1 -- 172.21.15.12:0/3858241788 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f22dc02f2e0 con 0x7f22e815e700 2024-04-15T15:22:19.677 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22d406c8f0 msgr2=0x7f22d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22d406c8f0 0x7f22d406edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f22e0003e60 tx=0x7f22e0009040 comp rx=0 tx=0).stop 2024-04-15T15:22:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e815e700 msgr2=0x7f22e8160030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e815e700 0x7f22e8160030 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f22dc000c00 tx=0x7f22dc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 shutdown_connections 2024-04-15T15:22:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f22d406c8f0 0x7f22d406edb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22e815eae0 0x7f22e80cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22e815e700 0x7f22e8160030 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 --2- 172.21.15.12:0/3858241788 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f22e80d2360 0x7f22e80ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 >> 172.21.15.12:0/3858241788 conn(0x7f22e808bf60 msgr2=0x7f22e80d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 shutdown_connections 2024-04-15T15:22:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.677+0000 7f22ef3bd640 1 -- 172.21.15.12:0/3858241788 wait complete. 2024-04-15T15:22:19.690 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:22:19.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.795+0000 7f627d3d4640 1 Processor -- start 2024-04-15T15:22:19.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.795+0000 7f627d3d4640 1 -- start start 2024-04-15T15:22:19.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.795+0000 7f627d3d4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f627815ecb0 0x7f62780b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.795+0000 7f627d3d4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f62780b4350 0x7f62780b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.795+0000 7f627d3d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62780b1b60 0x7f62780ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.795+0000 7f627d3d4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f627815f090 con 0x7f627815ecb0 2024-04-15T15:22:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.795+0000 7f627d3d4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f62780cdf80 con 0x7f62780b4350 2024-04-15T15:22:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.795+0000 7f627d3d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f62780b10d0 con 0x7f62780b1b60 2024-04-15T15:22:19.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f6276ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f627815ecb0 0x7f62780b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f6276ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f627815ecb0 0x7f62780b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55798/0 (socket says 172.21.15.12:55798) 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f6276ffd640 1 -- 172.21.15.12:0/3982220178 learned_addr learned my addr 172.21.15.12:0/3982220178 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f62777fe640 1 --2- 172.21.15.12:0/3982220178 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62780b1b60 0x7f62780ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f6276ffd640 1 -- 172.21.15.12:0/3982220178 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62780b1b60 msgr2=0x7f62780ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f62767fc640 1 --2- 172.21.15.12:0/3982220178 >> v2:172.21.15.55:3300/0 conn(0x7f62780b4350 0x7f62780b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f6276ffd640 1 --2- 172.21.15.12:0/3982220178 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62780b1b60 0x7f62780ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f6276ffd640 1 -- 172.21.15.12:0/3982220178 >> v2:172.21.15.55:3300/0 conn(0x7f62780b4350 msgr2=0x7f62780b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f6276ffd640 1 --2- 172.21.15.12:0/3982220178 >> v2:172.21.15.55:3300/0 conn(0x7f62780b4350 0x7f62780b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f6276ffd640 1 -- 172.21.15.12:0/3982220178 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62780af920 con 0x7f627815ecb0 2024-04-15T15:22:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.796+0000 7f62767fc640 1 --2- 172.21.15.12:0/3982220178 >> v2:172.21.15.55:3300/0 conn(0x7f62780b4350 0x7f62780b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f6276ffd640 1 --2- 172.21.15.12:0/3982220178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627815ecb0 0x7f62780b0b90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f627806a050 tx=0x7f626802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4029b218e2314cfe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f6275ffb640 1 -- 172.21.15.12:0/3982220178 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268034070 con 0x7f627815ecb0 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f6275ffb640 1 -- 172.21.15.12:0/3982220178 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6268030860 con 0x7f627815ecb0 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f6275ffb640 1 -- 172.21.15.12:0/3982220178 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268030c30 con 0x7f627815ecb0 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 -- 172.21.15.12:0/3982220178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627815ecb0 msgr2=0x7f62780b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 --2- 172.21.15.12:0/3982220178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627815ecb0 0x7f62780b0b90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f627806a050 tx=0x7f626802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 -- 172.21.15.12:0/3982220178 shutdown_connections 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 --2- 172.21.15.12:0/3982220178 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62780b1b60 0x7f62780ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 --2- 172.21.15.12:0/3982220178 >> v2:172.21.15.55:3300/0 conn(0x7f62780b4350 0x7f62780b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 --2- 172.21.15.12:0/3982220178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f627815ecb0 0x7f62780b0b90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 -- 172.21.15.12:0/3982220178 >> 172.21.15.12:0/3982220178 conn(0x7f62780b6870 msgr2=0x7f62780b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 -- 172.21.15.12:0/3982220178 shutdown_connections 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.797+0000 7f627d3d4640 1 -- 172.21.15.12:0/3982220178 wait complete. 2024-04-15T15:22:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.799+0000 7f627d3d4640 1 Processor -- start 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.799+0000 7f627d3d4640 1 -- start start 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.799+0000 7f627d3d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62780b1b60 0x7f6278160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.799+0000 7f627d3d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62780b4350 0x7f62780cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.799+0000 7f627d3d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f627815ecb0 0x7f6278150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.799+0000 7f627d3d4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f62780b47c0 con 0x7f62780b4350 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.799+0000 7f627d3d4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f62780afee0 con 0x7f62780b1b60 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.799+0000 7f627d3d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f62781607a0 con 0x7f627815ecb0 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62780b4350 0x7f62780cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62780b4350 0x7f62780cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55802/0 (socket says 172.21.15.12:55802) 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 -- 172.21.15.12:0/2968760000 learned_addr learned my addr 172.21.15.12:0/2968760000 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 -- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f627815ecb0 msgr2=0x7f6278150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f627815ecb0 0x7f6278150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 -- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62780b1b60 msgr2=0x7f6278160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62780b1b60 0x7f6278160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 -- 172.21.15.12:0/2968760000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6278150730 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f62767fc640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62780b4350 0x7f62780cd840 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f626c00c7f0 tx=0x7f626c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f625f7fe640 1 -- 172.21.15.12:0/2968760000 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c015070 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f625f7fe640 1 -- 172.21.15.12:0/2968760000 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f626c007a30 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62780cdd80 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f625f7fe640 1 -- 172.21.15.12:0/2968760000 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c007e00 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.800+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6278160cb0 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.801+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62780689e0 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.802+0000 7f625f7fe640 1 -- 172.21.15.12:0/2968760000 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f626c018070 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.802+0000 7f625f7fe640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f626406c8f0 0x7f626406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.802+0000 7f625f7fe640 1 -- 172.21.15.12:0/2968760000 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f626c08bf10 con 0x7f62780b4350 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.803+0000 7f6276ffd640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f626406c8f0 0x7f626406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:19.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.803+0000 7f6276ffd640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f626406c8f0 0x7f626406edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6268036000 tx=0x7f6268030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:19.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:19.807+0000 7f625f7fe640 1 -- 172.21.15.12:0/2968760000 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f626c059f90 con 0x7f62780b4350 2024-04-15T15:22:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.031+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f627806c7c0 con 0x7f62780b4350 2024-04-15T15:22:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.032+0000 7f625f7fe640 1 -- 172.21.15.12:0/2968760000 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f626c059890 con 0x7f62780b4350 2024-04-15T15:22:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:20.034 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:33.113220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:22:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.033+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f626406c8f0 msgr2=0x7f626406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.033+0000 7f627d3d4640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f626406c8f0 0x7f626406edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6268036000 tx=0x7f6268030000 comp rx=0 tx=0).stop 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62780b4350 msgr2=0x7f62780cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62780b4350 0x7f62780cd840 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f626c00c7f0 tx=0x7f626c00cbf0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 shutdown_connections 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f627815ecb0 0x7f6278150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62780b4350 0x7f62780cd840 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f626406c8f0 0x7f626406edb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 --2- 172.21.15.12:0/2968760000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62780b1b60 0x7f6278160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 >> 172.21.15.12:0/2968760000 conn(0x7f62780b6870 msgr2=0x7f62780b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 shutdown_connections 2024-04-15T15:22:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.034+0000 7f627d3d4640 1 -- 172.21.15.12:0/2968760000 wait complete. 2024-04-15T15:22:20.047 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:20.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.150+0000 7f2caa02c640 1 Processor -- start 2024-04-15T15:22:20.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.150+0000 7f2caa02c640 1 -- start start 2024-04-15T15:22:20.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2caa02c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 0x7f2ca40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2caa02c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2ca40d2360 0x7f2ca40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2caa02c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ca415eae0 0x7f2ca40d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2caa02c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2ca40b53b0 con 0x7f2ca415eae0 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2caa02c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2ca40b47a0 con 0x7f2ca40d2360 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2caa02c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca40b4a80 con 0x7f2ca415e700 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2ca3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ca415eae0 0x7f2ca40d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2ca3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ca415eae0 0x7f2ca40d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55816/0 (socket says 172.21.15.12:55816) 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2ca3fff640 1 -- 172.21.15.12:0/3137477694 learned_addr learned my addr 172.21.15.12:0/3137477694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2ca2ffd640 1 --2- 172.21.15.12:0/3137477694 >> v2:172.21.15.55:3300/0 conn(0x7f2ca40d2360 0x7f2ca40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2ca37fe640 1 --2- 172.21.15.12:0/3137477694 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 0x7f2ca40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2ca3fff640 1 -- 172.21.15.12:0/3137477694 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 msgr2=0x7f2ca40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.151+0000 7f2ca3fff640 1 --2- 172.21.15.12:0/3137477694 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 0x7f2ca40b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.152+0000 7f2ca3fff640 1 -- 172.21.15.12:0/3137477694 >> v2:172.21.15.55:3300/0 conn(0x7f2ca40d2360 msgr2=0x7f2ca40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.152+0000 7f2ca3fff640 1 --2- 172.21.15.12:0/3137477694 >> v2:172.21.15.55:3300/0 conn(0x7f2ca40d2360 0x7f2ca40d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.152+0000 7f2ca3fff640 1 -- 172.21.15.12:0/3137477694 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca40b4d60 con 0x7f2ca415eae0 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.152+0000 7f2ca3fff640 1 --2- 172.21.15.12:0/3137477694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca40d30a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40d2100 tx=0x7f2c9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f93d2d525d0c70 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.152+0000 7f2ca27fc640 1 -- 172.21.15.12:0/3137477694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c02ca70 con 0x7f2ca415eae0 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.152+0000 7f2ca27fc640 1 -- 172.21.15.12:0/3137477694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c9c02cc10 con 0x7f2ca415eae0 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.152+0000 7f2ca27fc640 1 -- 172.21.15.12:0/3137477694 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c0343f0 con 0x7f2ca415eae0 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 -- 172.21.15.12:0/3137477694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 msgr2=0x7f2ca40d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 --2- 172.21.15.12:0/3137477694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca40d30a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40d2100 tx=0x7f2c9c02bed0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 -- 172.21.15.12:0/3137477694 shutdown_connections 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 --2- 172.21.15.12:0/3137477694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca40d30a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 --2- 172.21.15.12:0/3137477694 >> v2:172.21.15.55:3300/0 conn(0x7f2ca40d2360 0x7f2ca40d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 --2- 172.21.15.12:0/3137477694 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 0x7f2ca40b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 -- 172.21.15.12:0/3137477694 >> 172.21.15.12:0/3137477694 conn(0x7f2ca408bf60 msgr2=0x7f2ca40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:20.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 -- 172.21.15.12:0/3137477694 shutdown_connections 2024-04-15T15:22:20.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.153+0000 7f2caa02c640 1 -- 172.21.15.12:0/3137477694 wait complete. 2024-04-15T15:22:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.154+0000 7f2caa02c640 1 Processor -- start 2024-04-15T15:22:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.154+0000 7f2caa02c640 1 -- start start 2024-04-15T15:22:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2caa02c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca40d2360 0x7f2ca4152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2caa02c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 0x7f2ca40b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2caa02c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca41614d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2caa02c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca40b5d80 con 0x7f2ca415eae0 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2caa02c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca40b4450 con 0x7f2ca40d2360 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2caa02c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca40d1d40 con 0x7f2ca415e700 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2ca3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca41614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2ca3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca41614d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55828/0 (socket says 172.21.15.12:55828) 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2ca3fff640 1 -- 172.21.15.12:0/2836757041 learned_addr learned my addr 172.21.15.12:0/2836757041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2ca37fe640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca40d2360 0x7f2ca4152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2ca2ffd640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 0x7f2ca40b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2ca3fff640 1 -- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 msgr2=0x7f2ca40b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.155+0000 7f2ca3fff640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 0x7f2ca40b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2ca3fff640 1 -- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca40d2360 msgr2=0x7f2ca4152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2ca3fff640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca40d2360 0x7f2ca4152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2ca3fff640 1 -- 172.21.15.12:0/2836757041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca4152fb0 con 0x7f2ca415eae0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2ca37fe640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca40d2360 0x7f2ca4152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2ca3fff640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca41614d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009f40 tx=0x7f2c9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2c8bfff640 1 -- 172.21.15.12:0/2836757041 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c043020 con 0x7f2ca415eae0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2c8bfff640 1 -- 172.21.15.12:0/2836757041 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c9c03e450 con 0x7f2ca415eae0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2c8bfff640 1 -- 172.21.15.12:0/2836757041 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c009510 con 0x7f2ca415eae0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca40b7b60 con 0x7f2ca415eae0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.156+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca40b8120 con 0x7f2ca415eae0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.157+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca40689e0 con 0x7f2ca415eae0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.158+0000 7f2c8bfff640 1 -- 172.21.15.12:0/2836757041 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c9c005d10 con 0x7f2ca415eae0 2024-04-15T15:22:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.158+0000 7f2c8bfff640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c9006c920 0x7f2c9006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.158+0000 7f2c8bfff640 1 -- 172.21.15.12:0/2836757041 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f2c9c0ae840 con 0x7f2ca415eae0 2024-04-15T15:22:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.158+0000 7f2ca37fe640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c9006c920 0x7f2c9006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.159+0000 7f2ca37fe640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c9006c920 0x7f2c9006ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009b80 tx=0x7f2c94008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.162+0000 7f2c8bfff640 1 -- 172.21.15.12:0/2836757041 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c9c07c8c0 con 0x7f2ca415eae0 2024-04-15T15:22:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.347+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ca40ba500 con 0x7f2ca415eae0 2024-04-15T15:22:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.348+0000 7f2c8bfff640 1 -- 172.21.15.12:0/2836757041 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f2c9c07c1c0 con 0x7f2ca415eae0 2024-04-15T15:22:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:20.350 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c9006c920 msgr2=0x7f2c9006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c9006c920 0x7f2c9006ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009b80 tx=0x7f2c94008040 comp rx=0 tx=0).stop 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 msgr2=0x7f2ca41614d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca41614d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009f40 tx=0x7f2c9c004a40 comp rx=0 tx=0).stop 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 shutdown_connections 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca415eae0 0x7f2ca41614d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca415e700 0x7f2ca40b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c9006c920 0x7f2c9006ede0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 --2- 172.21.15.12:0/2836757041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca40d2360 0x7f2ca4152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 >> 172.21.15.12:0/2836757041 conn(0x7f2ca408bf60 msgr2=0x7f2ca40cfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 shutdown_connections 2024-04-15T15:22:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.350+0000 7f2caa02c640 1 -- 172.21.15.12:0/2836757041 wait complete. 2024-04-15T15:22:20.363 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:22:20.364 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:22:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.468+0000 7fbcb1dbd640 1 Processor -- start 2024-04-15T15:22:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.468+0000 7fbcb1dbd640 1 -- start start 2024-04-15T15:22:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcb1dbd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbcac15ecb0 0x7fbcac0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcb1dbd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcac0b10d0 con 0x7fbcac0b1d30 2024-04-15T15:22:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcb1dbd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcac0af920 con 0x7fbcac0b4350 2024-04-15T15:22:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcb1dbd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fbcac15f090 con 0x7fbcac15ecb0 2024-04-15T15:22:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcabfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbcac0b1d30 0x7fbcac0ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50968/0 (socket says 172.21.15.12:50968) 2024-04-15T15:22:20.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcabfff640 1 -- 172.21.15.12:0/1424185726 learned_addr learned my addr 172.21.15.12:0/1424185726 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:20.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcab7fe640 1 --2- 172.21.15.12:0/1424185726 >> v2:172.21.15.55:3300/0 conn(0x7fbcac15ecb0 0x7fbcac0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcaa7fc640 1 -- 172.21.15.12:0/1424185726 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1733610309 0 0) 0x7fbcac0b10d0 con 0x7fbcac0b1d30 2024-04-15T15:22:20.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcaa7fc640 1 -- 172.21.15.12:0/1424185726 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc94003520 con 0x7fbcac0b1d30 2024-04-15T15:22:20.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcab7fe640 1 -- 172.21.15.12:0/1424185726 >> v1:172.21.15.55:6789/0 conn(0x7fbcac0b4350 legacy=0x7fbcac0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcab7fe640 1 -- 172.21.15.12:0/1424185726 >> v1:172.21.15.12:6789/0 conn(0x7fbcac0b1d30 legacy=0x7fbcac0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.469+0000 7fbcab7fe640 1 -- 172.21.15.12:0/1424185726 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac0afc00 con 0x7fbcac15ecb0 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.470+0000 7fbcab7fe640 1 --2- 172.21.15.12:0/1424185726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 0x7fbcac0b0b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0097d0 tx=0x7fbc9c02b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14682928b32c3770 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.470+0000 7fbcaa7fc640 1 -- 172.21.15.12:0/1424185726 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c034070 con 0x7fbcac15ecb0 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.470+0000 7fbcaa7fc640 1 -- 172.21.15.12:0/1424185726 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9c030890 con 0x7fbcac15ecb0 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.470+0000 7fbcaa7fc640 1 -- 172.21.15.12:0/1424185726 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c030c40 con 0x7fbcac15ecb0 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.471+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/1424185726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 msgr2=0x7fbcac0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.471+0000 7fbcb1dbd640 1 --2- 172.21.15.12:0/1424185726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 0x7fbcac0b0b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0097d0 tx=0x7fbc9c02b9d0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.471+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/1424185726 shutdown_connections 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.471+0000 7fbcb1dbd640 1 --2- 172.21.15.12:0/1424185726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 0x7fbcac0b0b90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.471+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/1424185726 >> 172.21.15.12:0/1424185726 conn(0x7fbcac0b6870 msgr2=0x7fbcac0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.471+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/1424185726 shutdown_connections 2024-04-15T15:22:20.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.471+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/1424185726 wait complete. 2024-04-15T15:22:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.472+0000 7fbcb1dbd640 1 Processor -- start 2024-04-15T15:22:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.472+0000 7fbcb1dbd640 1 -- start start 2024-04-15T15:22:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcb1dbd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbcac0b1d30 0x7fbcac15fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcb1dbd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcac0b4350 0x7fbcac14ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcb1dbd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 0x7fbcac06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcb1dbd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac0b47c0 con 0x7fbcac0b4350 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcb1dbd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac0cdf80 con 0x7fbcac15ecb0 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcb1dbd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fbcac158190 con 0x7fbcac0b1d30 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcaaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcac0b4350 0x7fbcac14ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcaaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcac0b4350 0x7fbcac14ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55844/0 (socket says 172.21.15.12:55844) 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcaaffd640 1 -- 172.21.15.12:0/103377223 learned_addr learned my addr 172.21.15.12:0/103377223 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcabfff640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 0x7fbcac06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcab7fe640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbcac0b1d30 0x7fbcac15fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcaaffd640 1 -- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbcac0b1d30 msgr2=0x7fbcac15fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcaaffd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbcac0b1d30 0x7fbcac15fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcaaffd640 1 -- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 msgr2=0x7fbcac06f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcaaffd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 0x7fbcac06f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.473+0000 7fbcaaffd640 1 -- 172.21.15.12:0/103377223 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac1f5630 con 0x7fbcac0b4350 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.474+0000 7fbcaaffd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcac0b4350 0x7fbcac14ed50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbca000c400 tx=0x7fbca000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.474+0000 7fbc93fff640 1 -- 172.21.15.12:0/103377223 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00052d0 con 0x7fbcac0b4350 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.474+0000 7fbc93fff640 1 -- 172.21.15.12:0/103377223 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca0005490 con 0x7fbcac0b4350 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.474+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcac1f5910 con 0x7fbcac0b4350 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.474+0000 7fbc93fff640 1 -- 172.21.15.12:0/103377223 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0007260 con 0x7fbcac0b4350 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.474+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcac15f7d0 con 0x7fbcac0b4350 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.475+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcac15f9d0 con 0x7fbcac0b4350 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.475+0000 7fbc93fff640 1 -- 172.21.15.12:0/103377223 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbca0007400 con 0x7fbcac0b4350 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.476+0000 7fbc93fff640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbc9806c8f0 0x7fbc9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.476+0000 7fbc93fff640 1 -- 172.21.15.12:0/103377223 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7fbca008b500 con 0x7fbcac0b4350 2024-04-15T15:22:20.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.476+0000 7fbcab7fe640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbc9806c8f0 0x7fbc9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.477+0000 7fbcab7fe640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbc9806c8f0 0x7fbc9806edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c036000 tx=0x7fbc9c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.480+0000 7fbc93fff640 1 -- 172.21.15.12:0/103377223 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbcac15f9d0 con 0x7fbcac0b4350 2024-04-15T15:22:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.704+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcac160290 con 0x7fbcac0b4350 2024-04-15T15:22:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.704+0000 7fbc93fff640 1 -- 172.21.15.12:0/103377223 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7fbca0058e00 con 0x7fbcac0b4350 2024-04-15T15:22:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:20.706 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:33.113220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:22:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-15T15:22:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbc9806c8f0 msgr2=0x7fbc9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbc9806c8f0 0x7fbc9806edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c036000 tx=0x7fbc9c008030 comp rx=0 tx=0).stop 2024-04-15T15:22:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcac0b4350 msgr2=0x7fbcac14ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcac0b4350 0x7fbcac14ed50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbca000c400 tx=0x7fbca000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 shutdown_connections 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbc9806c8f0 0x7fbc9806edb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac15ecb0 0x7fbcac06f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbcac0b4350 0x7fbcac14ed50 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 --2- 172.21.15.12:0/103377223 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbcac0b1d30 0x7fbcac15fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 >> 172.21.15.12:0/103377223 conn(0x7fbcac0b6870 msgr2=0x7fbcac070460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 shutdown_connections 2024-04-15T15:22:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.706+0000 7fbcb1dbd640 1 -- 172.21.15.12:0/103377223 wait complete. 2024-04-15T15:22:20.719 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:22:20.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.823+0000 7f0adec93640 1 Processor -- start 2024-04-15T15:22:20.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0adec93640 1 -- start start 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0adec93640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad809a9d0 0x7f0ad8160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0adec93640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ad80701f0 0x7f0ad815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0adec93640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0ad809adb0 0x7f0ad80d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0adec93640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0ad809ba90 con 0x7f0ad80701f0 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0adec93640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0ad8160070 con 0x7f0ad809adb0 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0adec93640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad80d08c0 con 0x7f0ad809a9d0 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0ad7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ad80701f0 0x7f0ad815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.824+0000 7f0ad7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ad80701f0 0x7f0ad815fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55852/0 (socket says 172.21.15.12:55852) 2024-04-15T15:22:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.825+0000 7f0ad7fff640 1 -- 172.21.15.12:0/1930688529 learned_addr learned my addr 172.21.15.12:0/1930688529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.825+0000 7f0adca2f640 1 --2- 172.21.15.12:0/1930688529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad809a9d0 0x7f0ad8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.825+0000 7f0add230640 1 --2- 172.21.15.12:0/1930688529 >> v2:172.21.15.55:3300/0 conn(0x7f0ad809adb0 0x7f0ad80d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.825+0000 7f0ad7fff640 1 -- 172.21.15.12:0/1930688529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad809a9d0 msgr2=0x7f0ad8160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.825+0000 7f0ad7fff640 1 --2- 172.21.15.12:0/1930688529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad809a9d0 0x7f0ad8160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.825+0000 7f0ad7fff640 1 -- 172.21.15.12:0/1930688529 >> v2:172.21.15.55:3300/0 conn(0x7f0ad809adb0 msgr2=0x7f0ad80d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.825+0000 7f0ad7fff640 1 --2- 172.21.15.12:0/1930688529 >> v2:172.21.15.55:3300/0 conn(0x7f0ad809adb0 0x7f0ad80d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.825+0000 7f0ad7fff640 1 -- 172.21.15.12:0/1930688529 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad8070720 con 0x7f0ad80701f0 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0ad7fff640 1 --2- 172.21.15.12:0/1930688529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad80701f0 0x7f0ad815fb30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009870 tx=0x7f0ac402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a627a168cf0fd954 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0ad77fe640 1 -- 172.21.15.12:0/1930688529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4035020 con 0x7f0ad80701f0 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0ad77fe640 1 -- 172.21.15.12:0/1930688529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac4031640 con 0x7f0ad80701f0 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0ad77fe640 1 -- 172.21.15.12:0/1930688529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac402d040 con 0x7f0ad80701f0 2024-04-15T15:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 -- 172.21.15.12:0/1930688529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad80701f0 msgr2=0x7f0ad815fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 --2- 172.21.15.12:0/1930688529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad80701f0 0x7f0ad815fb30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009870 tx=0x7f0ac402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 -- 172.21.15.12:0/1930688529 shutdown_connections 2024-04-15T15:22:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 --2- 172.21.15.12:0/1930688529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad80701f0 0x7f0ad815fb30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 --2- 172.21.15.12:0/1930688529 >> v2:172.21.15.55:3300/0 conn(0x7f0ad809adb0 0x7f0ad80d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 --2- 172.21.15.12:0/1930688529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad809a9d0 0x7f0ad8160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 -- 172.21.15.12:0/1930688529 >> 172.21.15.12:0/1930688529 conn(0x7f0ad80d2460 msgr2=0x7f0ad80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 -- 172.21.15.12:0/1930688529 shutdown_connections 2024-04-15T15:22:20.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.826+0000 7f0adec93640 1 -- 172.21.15.12:0/1930688529 wait complete. 2024-04-15T15:22:20.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.827+0000 7f0adec93640 1 Processor -- start 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.828+0000 7f0adec93640 1 -- start start 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.828+0000 7f0adec93640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad80701f0 0x7f0ad80c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.828+0000 7f0adec93640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ad809a9d0 0x7f0ad80cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.828+0000 7f0adec93640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad809adb0 0x7f0ad80cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.828+0000 7f0adec93640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad8071050 con 0x7f0ad809adb0 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.828+0000 7f0adec93640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad8070d70 con 0x7f0ad809a9d0 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.828+0000 7f0adec93640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad80b1c60 con 0x7f0ad80701f0 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.828+0000 7f0add230640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad809adb0 0x7f0ad80cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0add230640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad809adb0 0x7f0ad80cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55860/0 (socket says 172.21.15.12:55860) 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0add230640 1 -- 172.21.15.12:0/1554912375 learned_addr learned my addr 172.21.15.12:0/1554912375 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0ad7fff640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ad809a9d0 0x7f0ad80cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0adca2f640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad80701f0 0x7f0ad80c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0add230640 1 -- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad80701f0 msgr2=0x7f0ad80c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0add230640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad80701f0 0x7f0ad80c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0add230640 1 -- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ad809a9d0 msgr2=0x7f0ad80cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0add230640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ad809a9d0 0x7f0ad80cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0add230640 1 -- 172.21.15.12:0/1554912375 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad8160da0 con 0x7f0ad809adb0 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0add230640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad809adb0 0x7f0ad80cb4f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0007e60 tx=0x7f0ad000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0ad57fa640 1 -- 172.21.15.12:0/1554912375 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0014420 con 0x7f0ad809adb0 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0ad57fa640 1 -- 172.21.15.12:0/1554912375 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad0019070 con 0x7f0ad809adb0 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad80c9b10 con 0x7f0ad809adb0 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0ad57fa640 1 -- 172.21.15.12:0/1554912375 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0014820 con 0x7f0ad809adb0 2024-04-15T15:22:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.829+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad8161320 con 0x7f0ad809adb0 2024-04-15T15:22:20.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.830+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad80689e0 con 0x7f0ad809adb0 2024-04-15T15:22:20.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.831+0000 7f0ad57fa640 1 -- 172.21.15.12:0/1554912375 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ad00149c0 con 0x7f0ad809adb0 2024-04-15T15:22:20.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.831+0000 7f0ad57fa640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0abc06c920 0x7f0abc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:20.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.831+0000 7f0ad57fa640 1 -- 172.21.15.12:0/1554912375 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f0ad008c6a0 con 0x7f0ad809adb0 2024-04-15T15:22:20.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.832+0000 7f0adca2f640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0abc06c920 0x7f0abc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:20.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.832+0000 7f0adca2f640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0abc06c920 0x7f0abc06ede0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0acc006fd0 tx=0x7f0acc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:20.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:20.836+0000 7f0ad57fa640 1 -- 172.21.15.12:0/1554912375 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ad0059fb0 con 0x7f0ad809adb0 2024-04-15T15:22:21.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.060+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ad80d3370 con 0x7f0ad809adb0 2024-04-15T15:22:21.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.060+0000 7f0ad57fa640 1 -- 172.21.15.12:0/1554912375 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f0ad005e2b0 con 0x7f0ad809adb0 2024-04-15T15:22:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:21.062 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3939466929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3939466929},{"type":"v1","addr":"172.21.15.55:6835","nonce":3939466929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/820466442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":820466442},{"type":"v1","addr":"172.21.15.12:6834","nonce":820466442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2383956801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":2383956801},{"type":"v1","addr":"172.21.15.12:6835","nonce":2383956801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:21:11.000401+0000","modified":"2024-04-15T15:21:33.113220+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/1379664958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1379664958},{"type":"v1","addr":"172.21.15.55:6837","nonce":1379664958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-04-15T15:22:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-15T15:22:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0abc06c920 msgr2=0x7f0abc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0abc06c920 0x7f0abc06ede0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0acc006fd0 tx=0x7f0acc008040 comp rx=0 tx=0).stop 2024-04-15T15:22:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad809adb0 msgr2=0x7f0ad80cb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad809adb0 0x7f0ad80cb4f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0007e60 tx=0x7f0ad000e4b0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 shutdown_connections 2024-04-15T15:22:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ad809adb0 0x7f0ad80cb4f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0abc06c920 0x7f0abc06ede0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ad809a9d0 0x7f0ad80cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 --2- 172.21.15.12:0/1554912375 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ad80701f0 0x7f0ad80c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 >> 172.21.15.12:0/1554912375 conn(0x7f0ad80d2460 msgr2=0x7f0ad806db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 shutdown_connections 2024-04-15T15:22:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.062+0000 7f0adec93640 1 -- 172.21.15.12:0/1554912375 wait complete. 2024-04-15T15:22:21.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:21.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.180+0000 7f67c41e2640 1 Processor -- start 2024-04-15T15:22:21.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.180+0000 7f67c41e2640 1 -- start start 2024-04-15T15:22:21.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c41e2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc15e790 0x7f67bc0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c41e2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67bc0d2480 con 0x7f67bc0cfd20 2024-04-15T15:22:21.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c41e2640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67bc0d2760 con 0x7f67bc0d1610 2024-04-15T15:22:21.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c41e2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc0d2db0 con 0x7f67bc15e790 2024-04-15T15:22:21.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c177d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f67bc0cfd20 0x7f67bc0d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50978/0 (socket says 172.21.15.12:50978) 2024-04-15T15:22:21.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c177d640 1 -- 172.21.15.12:0/143951248 learned_addr learned my addr 172.21.15.12:0/143951248 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:21.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174827444 0 0) 0x7f67bc0d2480 con 0x7f67bc0cfd20 2024-04-15T15:22:21.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c1f7e640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc15e790 0x7f67bc0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67a8003520 con 0x7f67bc0cfd20 2024-04-15T15:22:21.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2301386013 0 0) 0x7f67a8003520 con 0x7f67bc0cfd20 2024-04-15T15:22:21.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc15e790 msgr2=0x7f67bc0d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:21.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc15e790 0x7f67bc0d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 >> v1:172.21.15.55:6789/0 conn(0x7f67bc0d1610 legacy=0x7f67bc0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc0d2a40 con 0x7f67bc0cfd20 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f67ac001670 con 0x7f67bc0cfd20 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 >> v1:172.21.15.12:6789/0 conn(0x7f67bc0cfd20 legacy=0x7f67bc0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67a80039e0 0x7f67a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc15e790 0x7f67a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67a8004d40 0x7f67a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc0d2db0 con 0x7f67a8004d40 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f67a800ba70 con 0x7f67a80039e0 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.181+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f67a800bc70 con 0x7f67bc15e790 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.182+0000 7f67c277f640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67a8004d40 0x7f67a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.182+0000 7f67c1f7e640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67a80039e0 0x7f67a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.182+0000 7f67c177d640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc15e790 0x7f67a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.182+0000 7f67c277f640 1 -- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc15e790 msgr2=0x7f67a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.182+0000 7f67c277f640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc15e790 0x7f67a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.182+0000 7f67c277f640 1 -- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67a80039e0 msgr2=0x7f67a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.182+0000 7f67c277f640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67a80039e0 0x7f67a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.182+0000 7f67c277f640 1 -- 172.21.15.12:0/143951248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a8003520 con 0x7f67a8004d40 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c277f640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67a8004d40 0x7f67a80051c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f67b4001e20 tx=0x7f67a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd6778aff2b617f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4001a70 con 0x7f67a8004d40 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b4003b70 con 0x7f67a8004d40 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c0f7c640 1 -- 172.21.15.12:0/143951248 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b40050d0 con 0x7f67a8004d40 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c41e2640 1 -- 172.21.15.12:0/143951248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67a8004d40 msgr2=0x7f67a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c41e2640 1 --2- 172.21.15.12:0/143951248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67a8004d40 0x7f67a80051c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f67b4001e20 tx=0x7f67a8005830 comp rx=0 tx=0).stop 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c277f640 1 -- 172.21.15.12:0/143951248 reap_dead start 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c41e2640 1 -- 172.21.15.12:0/143951248 shutdown_connections 2024-04-15T15:22:21.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c41e2640 1 -- 172.21.15.12:0/143951248 >> 172.21.15.12:0/143951248 conn(0x7f67bc08bf60 msgr2=0x7f67bc0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:21.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c41e2640 1 -- 172.21.15.12:0/143951248 shutdown_connections 2024-04-15T15:22:21.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.183+0000 7f67c41e2640 1 -- 172.21.15.12:0/143951248 wait complete. 2024-04-15T15:22:21.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.185+0000 7f67c41e2640 1 Processor -- start 2024-04-15T15:22:21.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.185+0000 7f67c41e2640 1 -- start start 2024-04-15T15:22:21.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.185+0000 7f67c41e2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc1516c0 0x7f67bc15dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.185+0000 7f67c41e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67bc151c50 0x7f67bc153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.185+0000 7f67c41e2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc1537d0 0x7f67bc155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.185+0000 7f67c41e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc15ecf0 con 0x7f67bc151c50 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.185+0000 7f67c41e2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc15eb70 con 0x7f67bc1537d0 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.185+0000 7f67c41e2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc0d0160 con 0x7f67bc1516c0 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67bc151c50 0x7f67bc153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67bc151c50 0x7f67bc153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55892/0 (socket says 172.21.15.12:55892) 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 -- 172.21.15.12:0/794164821 learned_addr learned my addr 172.21.15.12:0/794164821 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c1f7e640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc1516c0 0x7f67bc15dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c277f640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc1537d0 0x7f67bc155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 -- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc1516c0 msgr2=0x7f67bc15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc1516c0 0x7f67bc15dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 -- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc1537d0 msgr2=0x7f67bc155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc1537d0 0x7f67bc155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 -- 172.21.15.12:0/794164821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc1521f0 con 0x7f67bc151c50 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c177d640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67bc151c50 0x7f67bc153290 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0072f0 tx=0x7f67ac0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67b27fc640 1 -- 172.21.15.12:0/794164821 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac00f9c0 con 0x7f67bc151c50 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67b27fc640 1 -- 172.21.15.12:0/794164821 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ac018070 con 0x7f67bc151c50 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67b27fc640 1 -- 172.21.15.12:0/794164821 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac00fdc0 con 0x7f67bc151c50 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc1524d0 con 0x7f67bc151c50 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.186+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc152a90 con 0x7f67bc151c50 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.187+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67bc0689e0 con 0x7f67bc151c50 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.188+0000 7f67b27fc640 1 -- 172.21.15.12:0/794164821 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67ac01d460 con 0x7f67bc151c50 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.188+0000 7f67b27fc640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67a006c8f0 0x7f67a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.188+0000 7f67b27fc640 1 -- 172.21.15.12:0/794164821 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f67ac08c190 con 0x7f67bc151c50 2024-04-15T15:22:21.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.188+0000 7f67c1f7e640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67a006c8f0 0x7f67a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.189+0000 7f67c1f7e640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67a006c8f0 0x7f67a006edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009b00 tx=0x7f67b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.193+0000 7f67b27fc640 1 -- 172.21.15.12:0/794164821 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67ac05a180 con 0x7f67bc151c50 2024-04-15T15:22:21.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.378+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67bc0b8310 con 0x7f67bc151c50 2024-04-15T15:22:21.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.379+0000 7f67b27fc640 1 -- 172.21.15.12:0/794164821 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18958 (secure 0 0 0) 0x7f67ac059a80 con 0x7f67bc151c50 2024-04-15T15:22:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:21.381 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:21:51.849099+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:21:08.716989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:21:09.952110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:21:13.669291+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67a006c8f0 msgr2=0x7f67a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67a006c8f0 0x7f67a006edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009b00 tx=0x7f67b8008040 comp rx=0 tx=0).stop 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67bc151c50 msgr2=0x7f67bc153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67bc151c50 0x7f67bc153290 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0072f0 tx=0x7f67ac0077b0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 shutdown_connections 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f67a006c8f0 0x7f67a006edb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc1537d0 0x7f67bc155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67bc151c50 0x7f67bc153290 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 --2- 172.21.15.12:0/794164821 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f67bc1516c0 0x7f67bc15dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 >> 172.21.15.12:0/794164821 conn(0x7f67bc08bf60 msgr2=0x7f67bc0ca180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 shutdown_connections 2024-04-15T15:22:21.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.381+0000 7f67c41e2640 1 -- 172.21.15.12:0/794164821 wait complete. 2024-04-15T15:22:21.395 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:22:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.500+0000 7f1e96b31640 1 Processor -- start 2024-04-15T15:22:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.500+0000 7f1e96b31640 1 -- start start 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e96b31640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e900d15e0 0x7f1e900d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e96b31640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1e900d4a40 0x7f1e900d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e96b31640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e9015eb70 con 0x7f1e9015e790 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e96b31640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1e900d3790 con 0x7f1e900d15e0 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e96b31640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1e900d45b0 con 0x7f1e900d4a40 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e8ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e900d15e0 0x7f1e900d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e8ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e900d15e0 0x7f1e900d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55898/0 (socket says 172.21.15.12:55898) 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e8ffff640 1 -- 172.21.15.12:0/2460103579 learned_addr learned my addr 172.21.15.12:0/2460103579 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e950ce640 1 --2- 172.21.15.12:0/2460103579 >> v2:172.21.15.55:3300/0 conn(0x7f1e900d4a40 0x7f1e900d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e8ffff640 1 -- 172.21.15.12:0/2460103579 >> v2:172.21.15.55:3300/0 conn(0x7f1e900d4a40 msgr2=0x7f1e900d4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e8ffff640 1 --2- 172.21.15.12:0/2460103579 >> v2:172.21.15.55:3300/0 conn(0x7f1e900d4a40 0x7f1e900d4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e8ffff640 1 -- 172.21.15.12:0/2460103579 >> v1:172.21.15.55:6789/0 conn(0x7f1e9015e790 legacy=0x7f1e900b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e8ffff640 1 -- 172.21.15.12:0/2460103579 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e900d3c80 con 0x7f1e900d15e0 2024-04-15T15:22:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.501+0000 7f1e950ce640 1 --2- 172.21.15.12:0/2460103579 >> v2:172.21.15.55:3300/0 conn(0x7f1e900d4a40 0x7f1e900d4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.502+0000 7f1e8ffff640 1 --2- 172.21.15.12:0/2460103579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d15e0 0x7f1e900d1a90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1e900b61c0 tx=0x7f1e7c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b8ab476f087bcc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.502+0000 7f1e8f7fe640 1 -- 172.21.15.12:0/2460103579 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c02e070 con 0x7f1e900d15e0 2024-04-15T15:22:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.502+0000 7f1e8f7fe640 1 -- 172.21.15.12:0/2460103579 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e7c037070 con 0x7f1e900d15e0 2024-04-15T15:22:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.502+0000 7f1e8f7fe640 1 -- 172.21.15.12:0/2460103579 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c031a00 con 0x7f1e900d15e0 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.503+0000 7f1e96b31640 1 -- 172.21.15.12:0/2460103579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d15e0 msgr2=0x7f1e900d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.503+0000 7f1e96b31640 1 --2- 172.21.15.12:0/2460103579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d15e0 0x7f1e900d1a90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1e900b61c0 tx=0x7f1e7c02b880 comp rx=0 tx=0).stop 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.503+0000 7f1e96b31640 1 -- 172.21.15.12:0/2460103579 shutdown_connections 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.503+0000 7f1e96b31640 1 --2- 172.21.15.12:0/2460103579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d15e0 0x7f1e900d1a90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.503+0000 7f1e96b31640 1 --2- 172.21.15.12:0/2460103579 >> v2:172.21.15.55:3300/0 conn(0x7f1e900d4a40 0x7f1e900d4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.503+0000 7f1e96b31640 1 -- 172.21.15.12:0/2460103579 >> 172.21.15.12:0/2460103579 conn(0x7f1e90067b40 msgr2=0x7f1e90067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.503+0000 7f1e96b31640 1 -- 172.21.15.12:0/2460103579 shutdown_connections 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.503+0000 7f1e96b31640 1 -- 172.21.15.12:0/2460103579 wait complete. 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.504+0000 7f1e96b31640 1 Processor -- start 2024-04-15T15:22:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.504+0000 7f1e96b31640 1 -- start start 2024-04-15T15:22:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e96b31640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e900d15e0 0x7f1e900ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e96b31640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d4a40 0x7f1e901509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e96b31640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e9015e790 0x7f1e900ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e96b31640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1e900d4240 con 0x7f1e900d4a40 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e96b31640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1e900d53f0 con 0x7f1e9015e790 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e96b31640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1e900b7db0 con 0x7f1e900d15e0 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d4a40 0x7f1e901509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d4a40 0x7f1e901509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55902/0 (socket says 172.21.15.12:55902) 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 -- 172.21.15.12:0/4151769847 learned_addr learned my addr 172.21.15.12:0/4151769847 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e950ce640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e9015e790 0x7f1e900ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e948cd640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e900d15e0 0x7f1e900ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 -- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e900d15e0 msgr2=0x7f1e900ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e900d15e0 0x7f1e900ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 -- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e9015e790 msgr2=0x7f1e900ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e9015e790 0x7f1e900ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 -- 172.21.15.12:0/4151769847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e90150f20 con 0x7f1e900d4a40 2024-04-15T15:22:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e948cd640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e900d15e0 0x7f1e900ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e950ce640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e9015e790 0x7f1e900ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.505+0000 7f1e8ffff640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d4a40 0x7f1e901509e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c02b8c0 tx=0x7f1e7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.506+0000 7f1e8d7fa640 1 -- 172.21.15.12:0/4151769847 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c032880 con 0x7f1e900d4a40 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.506+0000 7f1e8d7fa640 1 -- 172.21.15.12:0/4151769847 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e7c032a40 con 0x7f1e900d4a40 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.506+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e900cbaa0 con 0x7f1e900d4a40 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.506+0000 7f1e8d7fa640 1 -- 172.21.15.12:0/4151769847 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c032e30 con 0x7f1e900d4a40 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.506+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e900cc060 con 0x7f1e900d4a40 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.507+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e900689e0 con 0x7f1e900d4a40 2024-04-15T15:22:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.507+0000 7f1e8d7fa640 1 -- 172.21.15.12:0/4151769847 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e7c0453f0 con 0x7f1e900d4a40 2024-04-15T15:22:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.508+0000 7f1e8d7fa640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e7406c8f0 0x7f1e7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.508+0000 7f1e8d7fa640 1 -- 172.21.15.12:0/4151769847 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6793+0+0 (secure 0 0 0) 0x7f1e7c0adff0 con 0x7f1e900d4a40 2024-04-15T15:22:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.508+0000 7f1e948cd640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e7406c8f0 0x7f1e7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.508+0000 7f1e948cd640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e7406c8f0 0x7f1e7406edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1e84007fd0 tx=0x7f1e84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.512+0000 7f1e8d7fa640 1 -- 172.21.15.12:0/4151769847 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e7c040070 con 0x7f1e900d4a40 2024-04-15T15:22:21.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.735+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1e900d4eb0 con 0x7f1e900d4a40 2024-04-15T15:22:21.737 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-04-15T15:22:21.736+0000 7fbdbcac3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:22:21.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.737+0000 7f1e8d7fa640 1 -- 172.21.15.12:0/4151769847 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7f1e7c080e20 con 0x7f1e900d4a40 2024-04-15T15:22:21.739 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:22:21.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.739+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e7406c8f0 msgr2=0x7f1e7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.739+0000 7f1e96b31640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e7406c8f0 0x7f1e7406edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1e84007fd0 tx=0x7f1e84009040 comp rx=0 tx=0).stop 2024-04-15T15:22:21.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.739+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d4a40 msgr2=0x7f1e901509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.739+0000 7f1e96b31640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d4a40 0x7f1e901509e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c02b8c0 tx=0x7f1e7c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.740+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 shutdown_connections 2024-04-15T15:22:21.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.740+0000 7f1e96b31640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e7406c8f0 0x7f1e7406edb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.740+0000 7f1e96b31640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e9015e790 0x7f1e900ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.740+0000 7f1e96b31640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e900d4a40 0x7f1e901509e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.740+0000 7f1e96b31640 1 --2- 172.21.15.12:0/4151769847 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e900d15e0 0x7f1e900ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.740+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 >> 172.21.15.12:0/4151769847 conn(0x7f1e90067b40 msgr2=0x7f1e900b9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:21.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.740+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 shutdown_connections 2024-04-15T15:22:21.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.740+0000 7f1e96b31640 1 -- 172.21.15.12:0/4151769847 wait complete. 2024-04-15T15:22:21.745 INFO:tasks.ceph.mds.d.smithi055.stderr: -4750> 2024-04-15T15:21:15.166+0000 7f06bd4b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:22:21.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-15T15:22:21.778 INFO:tasks.ceph.mds.d.smithi055.stdout:starting mds.d at 2024-04-15T15:22:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.858+0000 7f62e9474640 1 Processor -- start 2024-04-15T15:22:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.858+0000 7f62e9474640 1 -- start start 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e9474640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f62e4072690 0x7f62e415f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e9474640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e40d0690 0x7f62e40d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e9474640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62e406e6a0 con 0x7f62e40b5050 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e9474640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f62e406ec60 con 0x7f62e4072690 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e9474640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f62e406e440 con 0x7f62e40d0690 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e40d0690 0x7f62e40d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e2ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f62e4072690 0x7f62e415f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e40d0690 0x7f62e40d0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:36914/0 (socket says 172.21.15.12:36914) 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e2ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f62e4072690 0x7f62e415f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:41686/0 (socket says 172.21.15.12:41686) 2024-04-15T15:22:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e37fe640 1 -- 172.21.15.12:0/665157845 learned_addr learned my addr 172.21.15.12:0/665157845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e37fe640 1 -- 172.21.15.12:0/665157845 >> v2:172.21.15.55:3300/0 conn(0x7f62e4072690 msgr2=0x7f62e415f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e37fe640 1 --2- 172.21.15.12:0/665157845 >> v2:172.21.15.55:3300/0 conn(0x7f62e4072690 0x7f62e415f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.859+0000 7f62e37fe640 1 -- 172.21.15.12:0/665157845 >> v1:172.21.15.55:6789/0 conn(0x7f62e40b5050 legacy=0x7f62e406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.860+0000 7f62e37fe640 1 -- 172.21.15.12:0/665157845 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e406e980 con 0x7f62e40d0690 2024-04-15T15:22:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.860+0000 7f62e37fe640 1 --2- 172.21.15.12:0/665157845 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e40d0690 0x7f62e40d0ac0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009e50 tx=0x7f62dc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3c159a126c4b53c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.860+0000 7f62e27fc640 1 -- 172.21.15.12:0/665157845 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc02cb00 con 0x7f62e40d0690 2024-04-15T15:22:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e27fc640 1 -- 172.21.15.12:0/665157845 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc02cca0 con 0x7f62e40d0690 2024-04-15T15:22:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e27fc640 1 -- 172.21.15.12:0/665157845 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc034610 con 0x7f62e40d0690 2024-04-15T15:22:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e9474640 1 -- 172.21.15.12:0/665157845 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e40d0690 msgr2=0x7f62e40d0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e9474640 1 --2- 172.21.15.12:0/665157845 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e40d0690 0x7f62e40d0ac0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009e50 tx=0x7f62dc02bf60 comp rx=0 tx=0).stop 2024-04-15T15:22:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e9474640 1 -- 172.21.15.12:0/665157845 shutdown_connections 2024-04-15T15:22:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e9474640 1 --2- 172.21.15.12:0/665157845 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e40d0690 0x7f62e40d0ac0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e9474640 1 --2- 172.21.15.12:0/665157845 >> v2:172.21.15.55:3300/0 conn(0x7f62e4072690 0x7f62e415f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e9474640 1 -- 172.21.15.12:0/665157845 >> 172.21.15.12:0/665157845 conn(0x7f62e406be00 msgr2=0x7f62e406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e9474640 1 -- 172.21.15.12:0/665157845 shutdown_connections 2024-04-15T15:22:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.861+0000 7f62e9474640 1 -- 172.21.15.12:0/665157845 wait complete. 2024-04-15T15:22:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.862+0000 7f62e9474640 1 Processor -- start 2024-04-15T15:22:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.862+0000 7f62e9474640 1 -- start start 2024-04-15T15:22:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62e9474640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e4072690 0x7f62e415e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62e9474640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62e40b5050 0x7f62e415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62e9474640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62e40d0690 0x7f62e414d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62e9474640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f62e415f160 con 0x7f62e40b5050 2024-04-15T15:22:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62e9474640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f62e40d1000 con 0x7f62e40d0690 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62e9474640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f62e415fbc0 con 0x7f62e4072690 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62e40b5050 0x7f62e415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62e40b5050 0x7f62e415ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55918/0 (socket says 172.21.15.12:55918) 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62dbfff640 1 -- 172.21.15.12:0/4206726736 learned_addr learned my addr 172.21.15.12:0/4206726736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62e2ffd640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e4072690 0x7f62e415e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62e37fe640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62e40d0690 0x7f62e414d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62dbfff640 1 -- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e4072690 msgr2=0x7f62e415e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62dbfff640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e4072690 0x7f62e415e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62dbfff640 1 -- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62e40d0690 msgr2=0x7f62e414d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62dbfff640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62e40d0690 0x7f62e414d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.863+0000 7f62dbfff640 1 -- 172.21.15.12:0/4206726736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e415d0d0 con 0x7f62e40b5050 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.864+0000 7f62dbfff640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62e40b5050 0x7f62e415ba90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00c460 tx=0x7f62cc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.864+0000 7f62db7fe640 1 -- 172.21.15.12:0/4206726736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc0052d0 con 0x7f62e40b5050 2024-04-15T15:22:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.864+0000 7f62db7fe640 1 -- 172.21.15.12:0/4206726736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62cc012630 con 0x7f62e40b5050 2024-04-15T15:22:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.864+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e415d3b0 con 0x7f62e40b5050 2024-04-15T15:22:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.864+0000 7f62db7fe640 1 -- 172.21.15.12:0/4206726736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc0054b0 con 0x7f62e40b5050 2024-04-15T15:22:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.864+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e415d970 con 0x7f62e40b5050 2024-04-15T15:22:21.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.865+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62e40689e0 con 0x7f62e40b5050 2024-04-15T15:22:21.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.865+0000 7f62db7fe640 1 -- 172.21.15.12:0/4206726736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f62cc012a20 con 0x7f62e40b5050 2024-04-15T15:22:21.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.866+0000 7f62db7fe640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f62c406c8f0 0x7f62c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:21.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.866+0000 7f62db7fe640 1 -- 172.21.15.12:0/4206726736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6879+0+0 (secure 0 0 0) 0x7f62cc08bb40 con 0x7f62e40b5050 2024-04-15T15:22:21.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.866+0000 7f62e2ffd640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f62c406c8f0 0x7f62c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:21.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.867+0000 7f62e2ffd640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f62c406c8f0 0x7f62c406edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f62e4157b90 tx=0x7f62d4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:21.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:21.871+0000 7f62db7fe640 1 -- 172.21.15.12:0/4206726736 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62cc059410 con 0x7f62e40b5050 2024-04-15T15:22:22.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.095+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f62e415ce10 con 0x7f62e40b5050 2024-04-15T15:22:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.739+0000 7f62db7fe640 1 -- 172.21.15.12:0/4206726736 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f62cc05d710 con 0x7f62e40b5050 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f62c406c8f0 msgr2=0x7f62c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f62c406c8f0 0x7f62c406edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f62e4157b90 tx=0x7f62d4009190 comp rx=0 tx=0).stop 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62e40b5050 msgr2=0x7f62e415ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62e40b5050 0x7f62e415ba90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00c460 tx=0x7f62cc00c920 comp rx=0 tx=0).stop 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 shutdown_connections 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f62c406c8f0 0x7f62c406edb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62e40d0690 0x7f62e414d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62e40b5050 0x7f62e415ba90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 --2- 172.21.15.12:0/4206726736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f62e4072690 0x7f62e415e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 >> 172.21.15.12:0/4206726736 conn(0x7f62e406be00 msgr2=0x7f62e406dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:22.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 shutdown_connections 2024-04-15T15:22:22.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.742+0000 7f62e9474640 1 -- 172.21.15.12:0/4206726736 wait complete. 2024-04-15T15:22:22.758 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-15T15:22:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.874+0000 7fa8ca3cd640 1 Processor -- start 2024-04-15T15:22:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.874+0000 7fa8ca3cd640 1 -- start start 2024-04-15T15:22:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8ca3cd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa8c415eb70 0x7fa8c40d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8ca3cd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8ca3cd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c40d2c00 con 0x7fa8c40d1070 2024-04-15T15:22:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8ca3cd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c40d2ee0 con 0x7fa8c415eb70 2024-04-15T15:22:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8ca3cd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c40d31c0 con 0x7fa8c40cfe80 2024-04-15T15:22:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8c3fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa8c415eb70 0x7fa8c40d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8c37fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8c896a640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa8c40d1070 0x7fa8c40b7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:58002/0 (socket says 172.21.15.12:58002) 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.875+0000 7fa8c896a640 1 -- 172.21.15.12:0/437626554 learned_addr learned my addr 172.21.15.12:0/437626554 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.876+0000 7fa8c37fe640 1 -- 172.21.15.12:0/437626554 >> v2:172.21.15.55:3300/0 conn(0x7fa8c415eb70 msgr2=0x7fa8c40d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.876+0000 7fa8c37fe640 1 --2- 172.21.15.12:0/437626554 >> v2:172.21.15.55:3300/0 conn(0x7fa8c415eb70 0x7fa8c40d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.876+0000 7fa8c37fe640 1 -- 172.21.15.12:0/437626554 >> v1:172.21.15.55:6789/0 conn(0x7fa8c40d1070 legacy=0x7fa8c40b7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.876+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/437626554 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1513716241 0 0) 0x7fa8c40d2c00 con 0x7fa8c40d1070 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.876+0000 7fa8c37fe640 1 -- 172.21.15.12:0/437626554 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c40d34a0 con 0x7fa8c40cfe80 2024-04-15T15:22:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.876+0000 7fa8c3fff640 1 --2- 172.21.15.12:0/437626554 >> v2:172.21.15.55:3300/0 conn(0x7fa8c415eb70 0x7fa8c40d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:22.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.876+0000 7fa8c37fe640 1 --2- 172.21.15.12:0/437626554 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c406d0b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80097a0 tx=0x7fa8b802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13f85cebdd57f65b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/437626554 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8035020 con 0x7fa8c40cfe80 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/437626554 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b8031830 con 0x7fa8c40cfe80 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/437626554 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8031c00 con 0x7fa8c40cfe80 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/437626554 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 msgr2=0x7fa8c406d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/437626554 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c406d0b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80097a0 tx=0x7fa8b802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/437626554 shutdown_connections 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/437626554 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c406d0b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/437626554 >> v2:172.21.15.55:3300/0 conn(0x7fa8c415eb70 0x7fa8c40d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/437626554 >> 172.21.15.12:0/437626554 conn(0x7fa8c40b61d0 msgr2=0x7fa8c40b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/437626554 shutdown_connections 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.877+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/437626554 wait complete. 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.878+0000 7fa8ca3cd640 1 Processor -- start 2024-04-15T15:22:22.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.878+0000 7fa8ca3cd640 1 -- start start 2024-04-15T15:22:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8ca3cd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c4156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8ca3cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8c40d1070 0x7fa8c4160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8ca3cd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8c415eb70 0x7fa8c41532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8ca3cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c406d770 con 0x7fa8c40d1070 2024-04-15T15:22:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8ca3cd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c406d5f0 con 0x7fa8c415eb70 2024-04-15T15:22:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8ca3cd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c40d1500 con 0x7fa8c40cfe80 2024-04-15T15:22:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8c37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8c40d1070 0x7fa8c4160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:22.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8c37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8c40d1070 0x7fa8c4160240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42324/0 (socket says 172.21.15.12:42324) 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8c37fe640 1 -- 172.21.15.12:0/1246972428 learned_addr learned my addr 172.21.15.12:0/1246972428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8c896a640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8c415eb70 0x7fa8c41532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.879+0000 7fa8c3fff640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c4156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c37fe640 1 -- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 msgr2=0x7fa8c4156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c37fe640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c4156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c37fe640 1 -- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8c415eb70 msgr2=0x7fa8c41532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c37fe640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8c415eb70 0x7fa8c41532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c37fe640 1 -- 172.21.15.12:0/1246972428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c4160d20 con 0x7fa8c40d1070 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c37fe640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8c40d1070 0x7fa8c4160240 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8000c00 tx=0x7fa8b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c0ff9640 1 -- 172.21.15.12:0/1246972428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8035070 con 0x7fa8c40d1070 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c0ff9640 1 -- 172.21.15.12:0/1246972428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b8004230 con 0x7fa8c40d1070 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c4161000 con 0x7fa8c40d1070 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8c0ff9640 1 -- 172.21.15.12:0/1246972428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b803a630 con 0x7fa8c40d1070 2024-04-15T15:22:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.880+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c41615c0 con 0x7fa8c40d1070 2024-04-15T15:22:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.881+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c41618f0 con 0x7fa8c40d1070 2024-04-15T15:22:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.882+0000 7fa8c0ff9640 1 -- 172.21.15.12:0/1246972428 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa8b8038020 con 0x7fa8c40d1070 2024-04-15T15:22:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.882+0000 7fa8c0ff9640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa8b006c920 0x7fa8b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.882+0000 7fa8c0ff9640 1 -- 172.21.15.12:0/1246972428 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6879+0+0 (secure 0 0 0) 0x7fa8b80adaf0 con 0x7fa8c40d1070 2024-04-15T15:22:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.883+0000 7fa8c3fff640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa8b006c920 0x7fa8b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.883+0000 7fa8c3fff640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa8b006c920 0x7fa8b006ede0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009560 tx=0x7fa8b4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:22.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:22.887+0000 7fa8c0ff9640 1 -- 172.21.15.12:0/1246972428 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8b8040070 con 0x7fa8c40d1070 2024-04-15T15:22:23.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:23.071+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa8c40689e0 con 0x7fa8c40d1070 2024-04-15T15:22:24.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.024+0000 7fa8c0ff9640 1 -- 172.21.15.12:0/1246972428 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7fa8b807b440 con 0x7fa8c40d1070 2024-04-15T15:22:24.025 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa8b006c920 msgr2=0x7fa8b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa8b006c920 0x7fa8b006ede0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009560 tx=0x7fa8b4009190 comp rx=0 tx=0).stop 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8c40d1070 msgr2=0x7fa8c4160240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8c40d1070 0x7fa8c4160240 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8000c00 tx=0x7fa8b80046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 shutdown_connections 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa8b006c920 0x7fa8b006ede0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8c415eb70 0x7fa8c41532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8c40d1070 0x7fa8c4160240 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 --2- 172.21.15.12:0/1246972428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa8c40cfe80 0x7fa8c4156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 >> 172.21.15.12:0/1246972428 conn(0x7fa8c40b61d0 msgr2=0x7fa8c40b4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:24.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 shutdown_connections 2024-04-15T15:22:24.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.027+0000 7fa8ca3cd640 1 -- 172.21.15.12:0/1246972428 wait complete. 2024-04-15T15:22:24.043 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-15T15:22:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.162+0000 7f13883d4640 1 Processor -- start 2024-04-15T15:22:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.163+0000 7f13883d4640 1 -- start start 2024-04-15T15:22:24.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.163+0000 7f13883d4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f138015e8c0 0x7f13800d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:24.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.163+0000 7f13883d4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f13800d1590 0x7f138006c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:24.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f13883d4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13800d2340 con 0x7f13800cfcb0 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f13883d4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f138015eca0 con 0x7f138015e8c0 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f13883d4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f13800d2620 con 0x7f13800d1590 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f1386170640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f138015e8c0 0x7f13800d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f1386170640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f138015e8c0 0x7f13800d3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42330/0 (socket says 172.21.15.12:42330) 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f1386170640 1 -- 172.21.15.12:0/2125296744 learned_addr learned my addr 172.21.15.12:0/2125296744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f1386971640 1 --2- 172.21.15.12:0/2125296744 >> v2:172.21.15.55:3300/0 conn(0x7f13800d1590 0x7f138006c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f1386170640 1 -- 172.21.15.12:0/2125296744 >> v2:172.21.15.55:3300/0 conn(0x7f13800d1590 msgr2=0x7f138006c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f1386170640 1 --2- 172.21.15.12:0/2125296744 >> v2:172.21.15.55:3300/0 conn(0x7f13800d1590 0x7f138006c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f1386170640 1 -- 172.21.15.12:0/2125296744 >> v1:172.21.15.55:6789/0 conn(0x7f13800cfcb0 legacy=0x7f13800d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.164+0000 7f1386170640 1 -- 172.21.15.12:0/2125296744 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13800d2900 con 0x7f138015e8c0 2024-04-15T15:22:24.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.165+0000 7f1386170640 1 --2- 172.21.15.12:0/2125296744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f138015e8c0 0x7f13800d3100 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f13800d5680 tx=0x7f137c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1d12ee2117f358 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.165+0000 7f138516e640 1 -- 172.21.15.12:0/2125296744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c034070 con 0x7f138015e8c0 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.165+0000 7f138516e640 1 -- 172.21.15.12:0/2125296744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137c030860 con 0x7f138015e8c0 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.165+0000 7f138516e640 1 -- 172.21.15.12:0/2125296744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c030c30 con 0x7f138015e8c0 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.166+0000 7f13883d4640 1 -- 172.21.15.12:0/2125296744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f138015e8c0 msgr2=0x7f13800d3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.166+0000 7f13883d4640 1 --2- 172.21.15.12:0/2125296744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f138015e8c0 0x7f13800d3100 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f13800d5680 tx=0x7f137c02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.166+0000 7f13883d4640 1 -- 172.21.15.12:0/2125296744 shutdown_connections 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.166+0000 7f13883d4640 1 --2- 172.21.15.12:0/2125296744 >> v2:172.21.15.55:3300/0 conn(0x7f13800d1590 0x7f138006c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.166+0000 7f13883d4640 1 --2- 172.21.15.12:0/2125296744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f138015e8c0 0x7f13800d3100 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.166+0000 7f13883d4640 1 -- 172.21.15.12:0/2125296744 >> 172.21.15.12:0/2125296744 conn(0x7f138008bf60 msgr2=0x7f13800d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.166+0000 7f13883d4640 1 -- 172.21.15.12:0/2125296744 shutdown_connections 2024-04-15T15:22:24.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.166+0000 7f13883d4640 1 -- 172.21.15.12:0/2125296744 wait complete. 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.167+0000 7f13883d4640 1 Processor -- start 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.167+0000 7f13883d4640 1 -- start start 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f13883d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13800cfcb0 0x7f13800cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f13883d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13800d1590 0x7f13801553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f13883d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f138015e8c0 0x7f13800b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f13883d4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f13800d4f60 con 0x7f13800d1590 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f13883d4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f13800d4de0 con 0x7f13800cfcb0 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f13883d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f13800d0140 con 0x7f138015e8c0 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13800d1590 0x7f13801553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13800d1590 0x7f13801553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42338/0 (socket says 172.21.15.12:42338) 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 -- 172.21.15.12:0/3074553511 learned_addr learned my addr 172.21.15.12:0/3074553511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:24.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f1386170640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13800cfcb0 0x7f13800cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f1386971640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f138015e8c0 0x7f13800b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 -- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f138015e8c0 msgr2=0x7f13800b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f138015e8c0 0x7f13800b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 -- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13800cfcb0 msgr2=0x7f13800cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13800cfcb0 0x7f13800cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 -- 172.21.15.12:0/3074553511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13800b7630 con 0x7f13800d1590 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f138596f640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13800d1590 0x7f13801553a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f137000c400 tx=0x7f137000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.168+0000 7f1376ffd640 1 -- 172.21.15.12:0/3074553511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13700052d0 con 0x7f13800d1590 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.169+0000 7f1376ffd640 1 -- 172.21.15.12:0/3074553511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1370005490 con 0x7f13800d1590 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.169+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13800b7910 con 0x7f13800d1590 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.169+0000 7f1376ffd640 1 -- 172.21.15.12:0/3074553511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370007260 con 0x7f13800d1590 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.169+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13800b7ed0 con 0x7f13800d1590 2024-04-15T15:22:24.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.170+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13800689e0 con 0x7f13800d1590 2024-04-15T15:22:24.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.170+0000 7f1376ffd640 1 -- 172.21.15.12:0/3074553511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1370007400 con 0x7f13800d1590 2024-04-15T15:22:24.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.170+0000 7f1376ffd640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f136406c8f0 0x7f136406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:24.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.171+0000 7f1376ffd640 1 -- 172.21.15.12:0/3074553511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6441+0+0 (secure 0 0 0) 0x7f137008b220 con 0x7f13800d1590 2024-04-15T15:22:24.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.171+0000 7f1386170640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f136406c8f0 0x7f136406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:24.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.171+0000 7f1386170640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f136406c8f0 0x7f136406edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f137c036000 tx=0x7f137c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:24.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.175+0000 7f1376ffd640 1 -- 172.21.15.12:0/3074553511 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1370058db0 con 0x7f13800d1590 2024-04-15T15:22:24.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:24.363+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f138006d6c0 con 0x7f13800d1590 2024-04-15T15:22:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.030+0000 7f1376ffd640 1 -- 172.21.15.12:0/3074553511 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7f137005d0b0 con 0x7f13800d1590 2024-04-15T15:22:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.034+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f136406c8f0 msgr2=0x7f136406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.034+0000 7f13883d4640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f136406c8f0 0x7f136406edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f137c036000 tx=0x7f137c03b040 comp rx=0 tx=0).stop 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.034+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13800d1590 msgr2=0x7f13801553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.034+0000 7f13883d4640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13800d1590 0x7f13801553a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f137000c400 tx=0x7f137000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.035+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 shutdown_connections 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.035+0000 7f13883d4640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f138015e8c0 0x7f13800b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.035+0000 7f13883d4640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13800d1590 0x7f13801553a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.035+0000 7f13883d4640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f136406c8f0 0x7f136406edb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.035+0000 7f13883d4640 1 --2- 172.21.15.12:0/3074553511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13800cfcb0 0x7f13800cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.035+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 >> 172.21.15.12:0/3074553511 conn(0x7f138008bf60 msgr2=0x7f138006c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.037+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 shutdown_connections 2024-04-15T15:22:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.038+0000 7f13883d4640 1 -- 172.21.15.12:0/3074553511 wait complete. 2024-04-15T15:22:25.055 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-15T15:22:25.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.167+0000 7f2c125d5640 1 Processor -- start 2024-04-15T15:22:25.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.167+0000 7f2c125d5640 1 -- start start 2024-04-15T15:22:25.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.168+0000 7f2c125d5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 0x7f2c0c0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.168+0000 7f2c125d5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2c0c0b1520 0x7f2c0c09b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.168+0000 7f2c125d5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c0c0b1a20 con 0x7f2c0c1626e0 2024-04-15T15:22:25.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.168+0000 7f2c125d5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2c0c0d0d90 con 0x7f2c0c0b1520 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.168+0000 7f2c125d5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0c0d0fe0 con 0x7f2c0c15eb70 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.168+0000 7f2c10b72640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2c0c1626e0 0x7f2c0c06d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60676/0 (socket says 172.21.15.12:60676) 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.168+0000 7f2c10b72640 1 -- 172.21.15.12:0/1059900245 learned_addr learned my addr 172.21.15.12:0/1059900245 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/1059900245 >> v2:172.21.15.55:3300/0 conn(0x7f2c0c0b1520 0x7f2c0c09b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 0x7f2c0c0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3308916660 0 0) 0x7f2c0c0b1a20 con 0x7f2c0c1626e0 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bf4003520 con 0x7f2c0c1626e0 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2324777124 0 0) 0x7f2bf4003520 con 0x7f2c0c1626e0 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 msgr2=0x7f2c0c0b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 0x7f2c0c0b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 >> v2:172.21.15.55:3300/0 conn(0x7f2c0c0b1520 msgr2=0x7f2c0c09b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 --2- 172.21.15.12:0/1059900245 >> v2:172.21.15.55:3300/0 conn(0x7f2c0c0b1520 0x7f2c0c09b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c0d12c0 con 0x7f2c0c1626e0 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f2c040019f0 con 0x7f2c0c1626e0 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 >> v1:172.21.15.12:6789/0 conn(0x7f2c0c1626e0 legacy=0x7f2c0c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 0x7f2bf40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bf4003f30 0x7f2bf4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4004d40 0x7f2bf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c0d0d90 con 0x7f2bf4003f30 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c0d0fe0 con 0x7f2bf4004d40 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2bf400ba10 con 0x7f2c0c15eb70 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bf4003f30 0x7f2bf4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 0x7f2bf40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c10b72640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4004d40 0x7f2bf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0b7fe640 1 -- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 msgr2=0x7f2bf40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 0x7f2bf40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0b7fe640 1 -- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4004d40 msgr2=0x7f2bf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4004d40 0x7f2bf40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0b7fe640 1 -- 172.21.15.12:0/1059900245 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf4003520 con 0x7f2bf4003f30 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.169+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c15eb70 0x7f2bf40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.170+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bf4003f30 0x7f2bf4004800 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2c00009af0 tx=0x7f2bf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4beda0be8bf3470c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.170+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c00039070 con 0x7f2bf4003f30 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.170+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c0002c710 con 0x7f2bf4003f30 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.170+0000 7f2c0affd640 1 -- 172.21.15.12:0/1059900245 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c00009690 con 0x7f2bf4003f30 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.171+0000 7f2c125d5640 1 -- 172.21.15.12:0/1059900245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bf4003f30 msgr2=0x7f2bf4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.171+0000 7f2c125d5640 1 --2- 172.21.15.12:0/1059900245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bf4003f30 0x7f2bf4004800 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2c00009af0 tx=0x7f2bf4005830 comp rx=0 tx=0).stop 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.171+0000 7f2c10b72640 1 -- 172.21.15.12:0/1059900245 reap_dead start 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.171+0000 7f2c125d5640 1 -- 172.21.15.12:0/1059900245 shutdown_connections 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.171+0000 7f2c125d5640 1 -- 172.21.15.12:0/1059900245 >> 172.21.15.12:0/1059900245 conn(0x7f2c0c0b3f70 msgr2=0x7f2c0c0b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.171+0000 7f2c125d5640 1 -- 172.21.15.12:0/1059900245 shutdown_connections 2024-04-15T15:22:25.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.171+0000 7f2c125d5640 1 -- 172.21.15.12:0/1059900245 wait complete. 2024-04-15T15:22:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.172+0000 7f2c125d5640 1 Processor -- start 2024-04-15T15:22:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.172+0000 7f2c125d5640 1 -- start start 2024-04-15T15:22:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c125d5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4003f30 0x7f2c0c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c125d5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c1e9560 0x7f2c0c1e8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c125d5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c1e8940 0x7f2c0c1e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c125d5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c09bd00 con 0x7f2c0c1e8940 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c125d5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c15ff60 con 0x7f2bf4003f30 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c125d5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0c0b4a90 con 0x7f2c0c1e9560 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c10b72640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c1e8940 0x7f2c0c1e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c10b72640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c1e8940 0x7f2c0c1e8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42366/0 (socket says 172.21.15.12:42366) 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c10b72640 1 -- 172.21.15.12:0/2925333007 learned_addr learned my addr 172.21.15.12:0/2925333007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4003f30 0x7f2c0c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c0b7fe640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c1e9560 0x7f2c0c1e8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c10b72640 1 -- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c1e9560 msgr2=0x7f2c0c1e8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c10b72640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c1e9560 0x7f2c0c1e8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c10b72640 1 -- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4003f30 msgr2=0x7f2c0c157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c10b72640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4003f30 0x7f2c0c157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.173+0000 7f2c10b72640 1 -- 172.21.15.12:0/2925333007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c0b28d0 con 0x7f2c0c1e8940 2024-04-15T15:22:25.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.174+0000 7f2c10b72640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c1e8940 0x7f2c0c1e8dc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0b3f20 tx=0x7f2c0400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.174+0000 7f2c08ff9640 1 -- 172.21.15.12:0/2925333007 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0400eda0 con 0x7f2c0c1e8940 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.174+0000 7f2c08ff9640 1 -- 172.21.15.12:0/2925333007 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c04018070 con 0x7f2c0c1e8940 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.174+0000 7f2c08ff9640 1 -- 172.21.15.12:0/2925333007 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04013660 con 0x7f2c0c1e8940 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.174+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c0c0b2bb0 con 0x7f2c0c1e8940 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.174+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c0c0b3170 con 0x7f2c0c1e8940 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.175+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c0c0689e0 con 0x7f2c0c1e8940 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.175+0000 7f2c08ff9640 1 -- 172.21.15.12:0/2925333007 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c04013800 con 0x7f2c0c1e8940 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.176+0000 7f2c08ff9640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2bf806c920 0x7f2bf806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.176+0000 7f2c08ff9640 1 -- 172.21.15.12:0/2925333007 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7f2c0408bd80 con 0x7f2c0c1e8940 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.176+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2bf806c920 0x7f2bf806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.177+0000 7f2c0bfff640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2bf806c920 0x7f2bf806ede0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009a60 tx=0x7f2bfc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.181+0000 7f2c08ff9640 1 -- 172.21.15.12:0/2925333007 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c04059b60 con 0x7f2c0c1e8940 2024-04-15T15:22:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:25.364+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2c0c0753c0 con 0x7f2c0c1e8940 2024-04-15T15:22:26.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.031+0000 7f2c08ff9640 1 -- 172.21.15.12:0/2925333007 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7f2c0405de60 con 0x7f2c0c1e8940 2024-04-15T15:22:26.033 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data_ec' removed 2024-04-15T15:22:26.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.034+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2bf806c920 msgr2=0x7f2bf806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:26.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.034+0000 7f2c125d5640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2bf806c920 0x7f2bf806ede0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009a60 tx=0x7f2bfc008040 comp rx=0 tx=0).stop 2024-04-15T15:22:26.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.034+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c1e8940 msgr2=0x7f2c0c1e8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:26.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.034+0000 7f2c125d5640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c1e8940 0x7f2c0c1e8dc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0b3f20 tx=0x7f2c0400e530 comp rx=0 tx=0).stop 2024-04-15T15:22:26.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.035+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 shutdown_connections 2024-04-15T15:22:26.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.035+0000 7f2c125d5640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c0c1e8940 0x7f2c0c1e8dc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:26.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.035+0000 7f2c125d5640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2c0c1e9560 0x7f2c0c1e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:26.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.035+0000 7f2c125d5640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2bf806c920 0x7f2bf806ede0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:26.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.035+0000 7f2c125d5640 1 --2- 172.21.15.12:0/2925333007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bf4003f30 0x7f2c0c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:26.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.035+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 >> 172.21.15.12:0/2925333007 conn(0x7f2c0c0b3f70 msgr2=0x7f2c0c074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:26.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.035+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 shutdown_connections 2024-04-15T15:22:26.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.035+0000 7f2c125d5640 1 -- 172.21.15.12:0/2925333007 wait complete. 2024-04-15T15:22:26.052 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-15T15:22:26.052 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-15T15:22:26.052 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-15T15:22:26.052 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:26.052 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-15T15:22:26.052 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-15T15:22:26.053 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-15T15:22:26.053 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:26.053 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-15T15:22:26.053 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-15T15:22:26.053 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-15T15:22:26.053 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:26.053 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-15T15:22:26.054 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-15T15:22:26.054 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-15T15:22:26.054 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:26.064 INFO:tasks.ceph.mds.c:Stopped 2024-04-15T15:22:26.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-15T15:22:26.065 INFO:tasks.ceph.mds.a:Stopped 2024-04-15T15:22:26.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-15T15:22:26.066 INFO:tasks.ceph.mds.b:Stopped 2024-04-15T15:22:26.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-15T15:22:26.067 INFO:tasks.ceph.mds.d:Stopped 2024-04-15T15:22:26.067 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-15T15:22:26.107 INFO:tasks.ceph.mds.c:Started 2024-04-15T15:22:26.107 DEBUG:teuthology.parallel:result is None 2024-04-15T15:22:26.108 INFO:tasks.ceph.mds.a:Started 2024-04-15T15:22:26.108 DEBUG:teuthology.parallel:result is None 2024-04-15T15:22:26.109 INFO:tasks.ceph.mds.b:Started 2024-04-15T15:22:26.109 DEBUG:teuthology.parallel:result is None 2024-04-15T15:22:26.110 INFO:tasks.ceph.mds.d:Started 2024-04-15T15:22:26.110 DEBUG:teuthology.parallel:result is None 2024-04-15T15:22:26.110 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-15T15:22:26.192 INFO:tasks.ceph.mds.a.smithi012.stdout:starting mds.a at 2024-04-15T15:22:26.192 INFO:tasks.ceph.mds.c.smithi012.stdout:starting mds.c at 2024-04-15T15:22:26.194 INFO:tasks.ceph.mds.b.smithi055.stdout:starting mds.b at 2024-04-15T15:22:26.194 INFO:tasks.ceph.mds.d.smithi055.stdout:starting mds.d at 2024-04-15T15:22:26.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.221+0000 7f0656488640 1 Processor -- start 2024-04-15T15:22:26.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.221+0000 7f0656488640 1 -- start start 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.222+0000 7f0656488640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06500d3270 0x7f06500cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.222+0000 7f0656488640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06500b47a0 con 0x7f065015e700 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.222+0000 7f0656488640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06500b4a80 con 0x7f06500d2360 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.222+0000 7f0656488640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f06500b53b0 con 0x7f06500d3270 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.222+0000 7f0654a25640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06500d3270 0x7f06500cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.222+0000 7f0654a25640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06500d3270 0x7f06500cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42400/0 (socket says 172.21.15.12:42400) 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.222+0000 7f0654a25640 1 -- 172.21.15.12:0/2430705701 learned_addr learned my addr 172.21.15.12:0/2430705701 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.223+0000 7f0654a25640 1 -- 172.21.15.12:0/2430705701 >> v1:172.21.15.55:6789/0 conn(0x7f06500d2360 legacy=0x7f06500d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.223+0000 7f0654a25640 1 -- 172.21.15.12:0/2430705701 >> v1:172.21.15.12:6789/0 conn(0x7f065015e700 legacy=0x7f06500b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:26.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.223+0000 7f0654a25640 1 -- 172.21.15.12:0/2430705701 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06500b4d60 con 0x7f06500d3270 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.223+0000 7f0654a25640 1 --2- 172.21.15.12:0/2430705701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d3270 0x7f06500cf370 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f06500b5530 tx=0x7f064802bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56816b55bb576831 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.223+0000 7f064effd640 1 -- 172.21.15.12:0/2430705701 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064802cb80 con 0x7f06500d3270 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.223+0000 7f064effd640 1 -- 172.21.15.12:0/2430705701 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f064802cd20 con 0x7f06500d3270 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.223+0000 7f064effd640 1 -- 172.21.15.12:0/2430705701 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648034610 con 0x7f06500d3270 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.224+0000 7f0656488640 1 -- 172.21.15.12:0/2430705701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d3270 msgr2=0x7f06500cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.224+0000 7f0656488640 1 --2- 172.21.15.12:0/2430705701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d3270 0x7f06500cf370 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f06500b5530 tx=0x7f064802bfe0 comp rx=0 tx=0).stop 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.224+0000 7f0656488640 1 -- 172.21.15.12:0/2430705701 shutdown_connections 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.224+0000 7f0656488640 1 --2- 172.21.15.12:0/2430705701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d3270 0x7f06500cf370 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.224+0000 7f0656488640 1 -- 172.21.15.12:0/2430705701 >> 172.21.15.12:0/2430705701 conn(0x7f065008bf60 msgr2=0x7f06500d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.224+0000 7f0656488640 1 -- 172.21.15.12:0/2430705701 shutdown_connections 2024-04-15T15:22:26.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.224+0000 7f0656488640 1 -- 172.21.15.12:0/2430705701 wait complete. 2024-04-15T15:22:26.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.225+0000 7f0656488640 1 Processor -- start 2024-04-15T15:22:26.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.225+0000 7f0656488640 1 -- start start 2024-04-15T15:22:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f0656488640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d2360 0x7f06500cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:26.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f0656488640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06500d3270 0x7f06500ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f0656488640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f065015e700 0x7f06501f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f0656488640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f06500d1d40 con 0x7f06500d2360 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f0656488640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f06500b4450 con 0x7f065015e700 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f0656488640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f06500d1ec0 con 0x7f06500d3270 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f064ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d2360 0x7f06500cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f064ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d2360 0x7f06500cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42410/0 (socket says 172.21.15.12:42410) 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.226+0000 7f064ffff640 1 -- 172.21.15.12:0/2262912087 learned_addr learned my addr 172.21.15.12:0/2262912087 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f0654a25640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f065015e700 0x7f06501f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064f7fe640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06500d3270 0x7f06500ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064ffff640 1 -- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06500d3270 msgr2=0x7f06500ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064ffff640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06500d3270 0x7f06500ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064ffff640 1 -- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f065015e700 msgr2=0x7f06501f73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064ffff640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f065015e700 0x7f06501f73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064ffff640 1 -- 172.21.15.12:0/2262912087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06500ccf90 con 0x7f06500d2360 2024-04-15T15:22:26.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064f7fe640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06500d3270 0x7f06500ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064ffff640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d2360 0x7f06500cc900 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f064000c400 tx=0x7f064000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064cff9640 1 -- 172.21.15.12:0/2262912087 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640017070 con 0x7f06500d2360 2024-04-15T15:22:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064cff9640 1 -- 172.21.15.12:0/2262912087 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06400035f0 con 0x7f06500d2360 2024-04-15T15:22:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06500cd270 con 0x7f06500d2360 2024-04-15T15:22:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f064cff9640 1 -- 172.21.15.12:0/2262912087 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06400039a0 con 0x7f06500d2360 2024-04-15T15:22:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.227+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06500cd830 con 0x7f06500d2360 2024-04-15T15:22:26.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.228+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06500753d0 con 0x7f06500d2360 2024-04-15T15:22:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.229+0000 7f064cff9640 1 -- 172.21.15.12:0/2262912087 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f064001a070 con 0x7f06500d2360 2024-04-15T15:22:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.229+0000 7f064cff9640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f063c06c8f0 0x7f063c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.229+0000 7f064cff9640 1 -- 172.21.15.12:0/2262912087 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7f064008abb0 con 0x7f06500d2360 2024-04-15T15:22:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.230+0000 7f064f7fe640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f063c06c8f0 0x7f063c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:26.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.230+0000 7f064f7fe640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f063c06c8f0 0x7f063c06edb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0644001680 tx=0x7f0644009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:26.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.234+0000 7f064cff9640 1 -- 172.21.15.12:0/2262912087 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06500753d0 con 0x7f06500d2360 2024-04-15T15:22:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:26.431+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f06500690f0 con 0x7f06500d2360 2024-04-15T15:22:27.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.072+0000 7f064cff9640 1 -- 172.21.15.12:0/2262912087 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7f06500690f0 con 0x7f06500d2360 2024-04-15T15:22:27.075 INFO:teuthology.orchestra.run.smithi012.stderr: removed all blocklist entries 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f063c06c8f0 msgr2=0x7f063c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f063c06c8f0 0x7f063c06edb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0644001680 tx=0x7f0644009040 comp rx=0 tx=0).stop 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d2360 msgr2=0x7f06500cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d2360 0x7f06500cc900 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f064000c400 tx=0x7f064000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 shutdown_connections 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f063c06c8f0 0x7f063c06edb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f065015e700 0x7f06501f73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f06500d3270 0x7f06500ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 --2- 172.21.15.12:0/2262912087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06500d2360 0x7f06500cc900 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 >> 172.21.15.12:0/2262912087 conn(0x7f065008bf60 msgr2=0x7f06500b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 shutdown_connections 2024-04-15T15:22:27.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.075+0000 7f0656488640 1 -- 172.21.15.12:0/2262912087 wait complete. 2024-04-15T15:22:27.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-15T15:22:27.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.196+0000 7fac4a255640 1 Processor -- start 2024-04-15T15:22:27.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.196+0000 7fac4a255640 1 -- start start 2024-04-15T15:22:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.197+0000 7fac4a255640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fac4415ecb0 0x7fac440b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.197+0000 7fac4a255640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac440b10d0 con 0x7fac440b4350 2024-04-15T15:22:27.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.197+0000 7fac4a255640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac440af920 con 0x7fac440b1d30 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.197+0000 7fac4a255640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fac4415f090 con 0x7fac4415ecb0 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.197+0000 7fac42ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fac440b4350 0x7fac440b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60712/0 (socket says 172.21.15.12:60712) 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.197+0000 7fac42ffd640 1 -- 172.21.15.12:0/2381328318 learned_addr learned my addr 172.21.15.12:0/2381328318 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.197+0000 7fac437fe640 1 --2- 172.21.15.12:0/2381328318 >> v2:172.21.15.55:3300/0 conn(0x7fac4415ecb0 0x7fac440b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 886733323 0 0) 0x7fac440b10d0 con 0x7fac440b4350 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac30003520 con 0x7fac440b4350 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2195740125 0 0) 0x7fac30003520 con 0x7fac440b4350 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 >> v2:172.21.15.55:3300/0 conn(0x7fac4415ecb0 msgr2=0x7fac440b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 --2- 172.21.15.12:0/2381328318 >> v2:172.21.15.55:3300/0 conn(0x7fac4415ecb0 0x7fac440b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 >> v1:172.21.15.55:6789/0 conn(0x7fac440b1d30 legacy=0x7fac440ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac440afc00 con 0x7fac440b4350 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fac2c001670 con 0x7fac440b4350 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 >> v1:172.21.15.12:6789/0 conn(0x7fac440b4350 legacy=0x7fac440b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac300039b0 0x7fac30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac30004800 0x7fac30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300051c0 0x7fac3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fac4415f090 con 0x7fac300039b0 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fac3000c070 con 0x7fac300051c0 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fac3000c270 con 0x7fac30004800 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac437fe640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac300039b0 0x7fac30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac43fff640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300051c0 0x7fac3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac42ffd640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac30004800 0x7fac30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac437fe640 1 -- 172.21.15.12:0/2381328318 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac30004800 msgr2=0x7fac30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac437fe640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac30004800 0x7fac30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac437fe640 1 -- 172.21.15.12:0/2381328318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300051c0 msgr2=0x7fac3000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:27.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac437fe640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300051c0 0x7fac3000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac437fe640 1 -- 172.21.15.12:0/2381328318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac30003520 con 0x7fac300039b0 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.198+0000 7fac43fff640 1 -- 172.21.15.12:0/2381328318 reap_dead start 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.199+0000 7fac437fe640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac300039b0 0x7fac30003dd0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fac34009fc0 tx=0x7fac30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff0221e7ce1d781 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.199+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac34039070 con 0x7fac300039b0 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.199+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3402cc20 con 0x7fac300039b0 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.199+0000 7fac427fc640 1 -- 172.21.15.12:0/2381328318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac34009c10 con 0x7fac300039b0 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.200+0000 7fac4a255640 1 -- 172.21.15.12:0/2381328318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac300039b0 msgr2=0x7fac30003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.200+0000 7fac4a255640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac300039b0 0x7fac30003dd0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fac34009fc0 tx=0x7fac30005830 comp rx=0 tx=0).stop 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.200+0000 7fac4a255640 1 -- 172.21.15.12:0/2381328318 shutdown_connections 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.200+0000 7fac4a255640 1 --2- 172.21.15.12:0/2381328318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac300039b0 0x7fac30003dd0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.200+0000 7fac4a255640 1 -- 172.21.15.12:0/2381328318 >> 172.21.15.12:0/2381328318 conn(0x7fac440b6870 msgr2=0x7fac440b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.200+0000 7fac4a255640 1 -- 172.21.15.12:0/2381328318 shutdown_connections 2024-04-15T15:22:27.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.200+0000 7fac4a255640 1 -- 172.21.15.12:0/2381328318 wait complete. 2024-04-15T15:22:27.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.201+0000 7fac4a255640 1 Processor -- start 2024-04-15T15:22:27.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.201+0000 7fac4a255640 1 -- start start 2024-04-15T15:22:27.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac4a255640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300039b0 0x7fac441f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac4a255640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac441578c0 0x7fac4415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac4a255640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac440ca6d0 0x7fac44161710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac4a255640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fac440b47c0 con 0x7fac441578c0 2024-04-15T15:22:27.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac4a255640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fac440cdf80 con 0x7fac300039b0 2024-04-15T15:22:27.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac4a255640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fac441f8be0 con 0x7fac440ca6d0 2024-04-15T15:22:27.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac441578c0 0x7fac4415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac441578c0 0x7fac4415ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42430/0 (socket says 172.21.15.12:42430) 2024-04-15T15:22:27.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac42ffd640 1 -- 172.21.15.12:0/52731954 learned_addr learned my addr 172.21.15.12:0/52731954 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac437fe640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300039b0 0x7fac441f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac42ffd640 1 -- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac440ca6d0 msgr2=0x7fac44161710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac43fff640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac440ca6d0 0x7fac44161710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac42ffd640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac440ca6d0 0x7fac44161710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac42ffd640 1 -- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300039b0 msgr2=0x7fac441f85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac42ffd640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300039b0 0x7fac441f85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac42ffd640 1 -- 172.21.15.12:0/52731954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac441f8ec0 con 0x7fac441578c0 2024-04-15T15:22:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.202+0000 7fac437fe640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300039b0 0x7fac441f85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.203+0000 7fac42ffd640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac441578c0 0x7fac4415ccc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00e830 tx=0x7fac2c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.203+0000 7fac3bfff640 1 -- 172.21.15.12:0/52731954 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00c8d0 con 0x7fac441578c0 2024-04-15T15:22:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.203+0000 7fac3bfff640 1 -- 172.21.15.12:0/52731954 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2c018070 con 0x7fac441578c0 2024-04-15T15:22:27.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.203+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac441f91a0 con 0x7fac441578c0 2024-04-15T15:22:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.203+0000 7fac3bfff640 1 -- 172.21.15.12:0/52731954 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00ccb0 con 0x7fac441578c0 2024-04-15T15:22:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.203+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac44157d40 con 0x7fac441578c0 2024-04-15T15:22:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.204+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac440b4110 con 0x7fac441578c0 2024-04-15T15:22:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.204+0000 7fac3bfff640 1 -- 172.21.15.12:0/52731954 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac2c01d460 con 0x7fac441578c0 2024-04-15T15:22:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.205+0000 7fac3bfff640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac2406c8f0 0x7fac2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.205+0000 7fac3bfff640 1 -- 172.21.15.12:0/52731954 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fac2c08b890 con 0x7fac441578c0 2024-04-15T15:22:27.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.205+0000 7fac437fe640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac2406c8f0 0x7fac2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.206+0000 7fac437fe640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac2406c8f0 0x7fac2406edb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fac34036000 tx=0x7fac3402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.209+0000 7fac3bfff640 1 -- 172.21.15.12:0/52731954 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac440b4110 con 0x7fac441578c0 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.464+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fac440b25f0 con 0x7fac441578c0 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.464+0000 7fac3bfff640 1 -- 172.21.15.12:0/52731954 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7fac440b25f0 con 0x7fac441578c0 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stdout: "auth_dump": [ 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.a", 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjRR1mfGBLNhAAzcTNRabmma9rGvlCj4vgFA==", 2024-04-15T15:22:27.467 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.b", 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCkRR1mncnxBxAAWy5Phi+MopzEoKYt7WlTXA==", 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.468 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.c", 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCkRR1m1fBGAhAAvTFwjNN+5kD3j429N9eQ/A==", 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mds.d", 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCkRR1mUtW6DxAAZKDeHi4Kmfhulx8nbIYepg==", 2024-04-15T15:22:27.469 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.0", 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCnRR1mDOuUMhAA+hU2FOSOsTeRWQXuSouJSw==", 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-04-15T15:22:27.470 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.471 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.471 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.1", 2024-04-15T15:22:27.471 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCqRR1m+puDMRAATMzipAvsvuQZXd04A5WtgA==", 2024-04-15T15:22:27.471 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.473 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-04-15T15:22:27.473 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-04-15T15:22:27.473 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.2", 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCtRR1m9qnfMBAA4rNHZSIpzmSqQ2RpWEQYdg==", 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.3", 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCwRR1m4r0ZMBAAOHHdTfsxkvI2I8i9HNbO1g==", 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.4", 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQC6RR1mwNH1CxAAkQq3vwEtpL7MdIpq+368QQ==", 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.5", 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQC9RR1mezNhDBAA4Py3odTAM0kR7yfal3jPCA==", 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.6", 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQDARR1m5GtZDBAAahBaCoZPZHsetB07+L/BEQ==", 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "osd.7", 2024-04-15T15:22:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQDDRR1m5hIsDBAAiUoogjGRfr6KeRjxU/9YTA==", 2024-04-15T15:22:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow profile osd", 2024-04-15T15:22:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile osd", 2024-04-15T15:22:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.0", 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCkRR1muiw8FBAAiEh+BBQRtVSuiPlO3ZezfA==", 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.1", 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCkRR1mQ4RTGxAAiJY6pHGO9TIYvUoQyoqwpw==", 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.2", 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCkRR1mWDLDFxAAGuwFUqNroSjT5XBZOXyONQ==", 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-04-15T15:22:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.3", 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCkRR1mZuxIIRAAOKEZ5qaFn2en0OyexnZX9A==", 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow", 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow r", 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow rw", 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow rwx" 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.admin", 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjRR1mOf/uFxAAJ/EUpanrBsnvnQV8WDf+eg==", 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": "allow *", 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow *", 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-mds", 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQDORR1mBbYpDhAA/8GX0MNrx0Ix1nJKrpQTqg==", 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-mds" 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-mgr", 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQDORR1maL4pDhAAJWMTeUyaKuw0wMuQ1JICng==", 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-osd", 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQDORR1mV8UpDhAAPwwMt/83dHT6apyd7uBAbA==", 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-osd" 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rbd", 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQDORR1mE8wpDhAA0fwf1zqT143i8euTp9Ftyg==", 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQDORR1m49IpDhAAvvvCxpDMIGTbm1jmFcD2Fg==", 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "client.bootstrap-rgw", 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQDORR1mi+0pDhAANznM8nKJKnYCWdRuZJEtVQ==", 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.x", 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjRR1mDPm2MhAAK0Z9rdK9KYolx2DI6EBTvw==", 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": "mgr.y", 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "AQCjRR1myBgtLhAA7md6R7R4ntQzqtiyaAh3iw==", 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": { 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "mds": "allow *", 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": "allow profile mgr", 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": "allow *" 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-15T15:22:27.487 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac2406c8f0 msgr2=0x7fac2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac2406c8f0 0x7fac2406edb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fac34036000 tx=0x7fac3402e000 comp rx=0 tx=0).stop 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac441578c0 msgr2=0x7fac4415ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac441578c0 0x7fac4415ccc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00e830 tx=0x7fac2c00ecf0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 shutdown_connections 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac440ca6d0 0x7fac44161710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac441578c0 0x7fac4415ccc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac2406c8f0 0x7fac2406edb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 --2- 172.21.15.12:0/52731954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300039b0 0x7fac441f85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 >> 172.21.15.12:0/52731954 conn(0x7fac440b6870 msgr2=0x7fac440b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:27.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 shutdown_connections 2024-04-15T15:22:27.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.466+0000 7fac4a255640 1 -- 172.21.15.12:0/52731954 wait complete. 2024-04-15T15:22:27.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-15T15:22:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.594+0000 7f1b596e4640 1 Processor -- start 2024-04-15T15:22:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.594+0000 7f1b596e4640 1 -- start start 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.594+0000 7f1b596e4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1b540d3240 0x7f1b540d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.594+0000 7f1b596e4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b540d2480 con 0x7f1b5415e790 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.594+0000 7f1b596e4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b540d2760 con 0x7f1b540d1440 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.594+0000 7f1b596e4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1b540d2db0 con 0x7f1b540d3240 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.594+0000 7f1b52ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1b5415e790 0x7f1b540cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60714/0 (socket says 172.21.15.12:60714) 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1342106647 learned_addr learned my addr 172.21.15.12:0/1342106647 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b527fc640 1 --2- 172.21.15.12:0/1342106647 >> v2:172.21.15.55:3300/0 conn(0x7f1b540d3240 0x7f1b540d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778524617 0 0) 0x7f1b540d2480 con 0x7f1b5415e790 2024-04-15T15:22:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b3c003520 con 0x7f1b5415e790 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2461308816 0 0) 0x7f1b3c003520 con 0x7f1b5415e790 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 >> v2:172.21.15.55:3300/0 conn(0x7f1b540d3240 msgr2=0x7f1b540d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 --2- 172.21.15.12:0/1342106647 >> v2:172.21.15.55:3300/0 conn(0x7f1b540d3240 0x7f1b540d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 >> v1:172.21.15.55:6789/0 conn(0x7f1b540d1440 legacy=0x7f1b540b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b540d2a40 con 0x7f1b5415e790 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f1b44001670 con 0x7f1b5415e790 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 >> v1:172.21.15.12:6789/0 conn(0x7f1b5415e790 legacy=0x7f1b540cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b3c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b3c0043d0 0x7f1b3c0048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b3c0050a0 0x7f1b3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b540d2db0 con 0x7f1b3c0039b0 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c00bfe0 con 0x7f1b3c0050a0 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3c00c1e0 con 0x7f1b3c0043d0 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b52ffd640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b3c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b527fc640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b3c0043d0 0x7f1b3c0048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b537fe640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b3c0050a0 0x7f1b3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b3c0043d0 msgr2=0x7f1b3c0048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b52ffd640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b3c0043d0 0x7f1b3c0048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b3c0050a0 msgr2=0x7f1b3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.595+0000 7f1b52ffd640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b3c0050a0 0x7f1b3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.596+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1342106647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c003520 con 0x7f1b3c0039b0 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.596+0000 7f1b537fe640 1 -- 172.21.15.12:0/1342106647 reap_dead start 2024-04-15T15:22:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.596+0000 7f1b527fc640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b3c0043d0 0x7f1b3c0048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.596+0000 7f1b52ffd640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b3c003db0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c240 tx=0x7f1b3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707db68d58c2e909 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.596+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44007d10 con 0x7f1b3c0039b0 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.596+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b44030070 con 0x7f1b3c0039b0 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.596+0000 7f1b51ffb640 1 -- 172.21.15.12:0/1342106647 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4402cda0 con 0x7f1b3c0039b0 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.597+0000 7f1b596e4640 1 -- 172.21.15.12:0/1342106647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 msgr2=0x7f1b3c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.597+0000 7f1b596e4640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b3c003db0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c240 tx=0x7f1b3c005830 comp rx=0 tx=0).stop 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.597+0000 7f1b596e4640 1 -- 172.21.15.12:0/1342106647 shutdown_connections 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.597+0000 7f1b596e4640 1 --2- 172.21.15.12:0/1342106647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b3c003db0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.597+0000 7f1b596e4640 1 -- 172.21.15.12:0/1342106647 >> 172.21.15.12:0/1342106647 conn(0x7f1b5408bf60 msgr2=0x7f1b540d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.597+0000 7f1b596e4640 1 -- 172.21.15.12:0/1342106647 shutdown_connections 2024-04-15T15:22:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.597+0000 7f1b596e4640 1 -- 172.21.15.12:0/1342106647 wait complete. 2024-04-15T15:22:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.598+0000 7f1b596e4640 1 Processor -- start 2024-04-15T15:22:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.598+0000 7f1b596e4640 1 -- start start 2024-04-15T15:22:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b596e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b5415f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b596e4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b54161970 0x7f1b54161db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b596e4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b5415a700 0x7f1b5415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b596e4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b540d5300 con 0x7f1b3c0039b0 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b596e4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b540d5180 con 0x7f1b5415a700 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b596e4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1b5415eb70 con 0x7f1b54161970 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b52ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b5415f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b52ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b5415f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42442/0 (socket says 172.21.15.12:42442) 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1965634919 learned_addr learned my addr 172.21.15.12:0/1965634919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b537fe640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b5415a700 0x7f1b5415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b527fc640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b54161970 0x7f1b54161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b54161970 msgr2=0x7f1b54161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b52ffd640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b54161970 0x7f1b54161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b5415a700 msgr2=0x7f1b5415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b52ffd640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b5415a700 0x7f1b5415ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1965634919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5415c0e0 con 0x7f1b3c0039b0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.599+0000 7f1b537fe640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b5415a700 0x7f1b5415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.600+0000 7f1b52ffd640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b5415f740 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c360 tx=0x7f1b540d2650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.600+0000 7f1b3b7fe640 1 -- 172.21.15.12:0/1965634919 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44001c50 con 0x7f1b3c0039b0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.600+0000 7f1b3b7fe640 1 -- 172.21.15.12:0/1965634919 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b44004bf0 con 0x7f1b3c0039b0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.600+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5415c3c0 con 0x7f1b3c0039b0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.600+0000 7f1b3b7fe640 1 -- 172.21.15.12:0/1965634919 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44007070 con 0x7f1b3c0039b0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.600+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5415c980 con 0x7f1b3c0039b0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.601+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5415cd60 con 0x7f1b3c0039b0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.601+0000 7f1b3b7fe640 1 -- 172.21.15.12:0/1965634919 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b44048020 con 0x7f1b3c0039b0 2024-04-15T15:22:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.602+0000 7f1b3b7fe640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b4006c920 0x7f1b4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.602+0000 7f1b3b7fe640 1 -- 172.21.15.12:0/1965634919 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f1b440acd20 con 0x7f1b3c0039b0 2024-04-15T15:22:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.602+0000 7f1b527fc640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b4006c920 0x7f1b4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.602+0000 7f1b527fc640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b4006c920 0x7f1b4006ede0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1b540ce740 tx=0x7f1b48008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.606+0000 7f1b3b7fe640 1 -- 172.21.15.12:0/1965634919 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b5415cd60 con 0x7f1b3c0039b0 2024-04-15T15:22:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.859+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f1b540689e0 con 0x7f1b3c0039b0 2024-04-15T15:22:27.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.861+0000 7f1b3b7fe640 1 -- 172.21.15.12:0/1965634919 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f1b44034e90 con 0x7f1b3c0039b0 2024-04-15T15:22:27.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b4006c920 msgr2=0x7f1b4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b4006c920 0x7f1b4006ede0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1b540ce740 tx=0x7f1b48008040 comp rx=0 tx=0).stop 2024-04-15T15:22:27.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 msgr2=0x7f1b5415f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b5415f740 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c360 tx=0x7f1b540d2650 comp rx=0 tx=0).stop 2024-04-15T15:22:27.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 shutdown_connections 2024-04-15T15:22:27.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1b4006c920 0x7f1b4006ede0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b5415a700 0x7f1b5415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1b54161970 0x7f1b54161db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 --2- 172.21.15.12:0/1965634919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b3c0039b0 0x7f1b5415f740 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 >> 172.21.15.12:0/1965634919 conn(0x7f1b5408bf60 msgr2=0x7f1b540d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:27.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 shutdown_connections 2024-04-15T15:22:27.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.863+0000 7f1b596e4640 1 -- 172.21.15.12:0/1965634919 wait complete. 2024-04-15T15:22:27.877 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-15T15:22:27.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.980+0000 7fdb18e74640 1 Processor -- start 2024-04-15T15:22:27.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.980+0000 7fdb18e74640 1 -- start start 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb18e74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb140ce8e0 0x7fdb14070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb18e74640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb14071d90 0x7fdb1409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb18e74640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdb14071240 0x7fdb1409af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb18e74640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fdb14070820 con 0x7fdb14071d90 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb18e74640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fdb1406ff30 con 0x7fdb14071240 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb18e74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fdb140cecc0 con 0x7fdb140ce8e0 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb0bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb14071d90 0x7fdb1409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb0bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb14071d90 0x7fdb1409a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42452/0 (socket says 172.21.15.12:42452) 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb0bfff640 1 -- 172.21.15.12:0/746290053 learned_addr learned my addr 172.21.15.12:0/746290053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb1259c640 1 --2- 172.21.15.12:0/746290053 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb140ce8e0 0x7fdb14070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb12d9d640 1 --2- 172.21.15.12:0/746290053 >> v2:172.21.15.55:3300/0 conn(0x7fdb14071240 0x7fdb1409af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb0bfff640 1 -- 172.21.15.12:0/746290053 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb140ce8e0 msgr2=0x7fdb14070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:27.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb0bfff640 1 --2- 172.21.15.12:0/746290053 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb140ce8e0 0x7fdb14070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb0bfff640 1 -- 172.21.15.12:0/746290053 >> v2:172.21.15.55:3300/0 conn(0x7fdb14071240 msgr2=0x7fdb1409af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:27.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb0bfff640 1 --2- 172.21.15.12:0/746290053 >> v2:172.21.15.55:3300/0 conn(0x7fdb14071240 0x7fdb1409af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.981+0000 7fdb0bfff640 1 -- 172.21.15.12:0/746290053 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1409b9d0 con 0x7fdb14071d90 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.982+0000 7fdb0bfff640 1 --2- 172.21.15.12:0/746290053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb14071d90 0x7fdb1409a9d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdb14161500 tx=0x7fdafc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d15327f410d3e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.982+0000 7fdb11d9b640 1 -- 172.21.15.12:0/746290053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc02e070 con 0x7fdb14071d90 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.982+0000 7fdb11d9b640 1 -- 172.21.15.12:0/746290053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdafc037070 con 0x7fdb14071d90 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.982+0000 7fdb11d9b640 1 -- 172.21.15.12:0/746290053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc031a00 con 0x7fdb14071d90 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 -- 172.21.15.12:0/746290053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb14071d90 msgr2=0x7fdb1409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 --2- 172.21.15.12:0/746290053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb14071d90 0x7fdb1409a9d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdb14161500 tx=0x7fdafc02b880 comp rx=0 tx=0).stop 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 -- 172.21.15.12:0/746290053 shutdown_connections 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 --2- 172.21.15.12:0/746290053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb14071d90 0x7fdb1409a9d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 --2- 172.21.15.12:0/746290053 >> v2:172.21.15.55:3300/0 conn(0x7fdb14071240 0x7fdb1409af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 --2- 172.21.15.12:0/746290053 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb140ce8e0 0x7fdb14070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 -- 172.21.15.12:0/746290053 >> 172.21.15.12:0/746290053 conn(0x7fdb140b3f40 msgr2=0x7fdb140b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:27.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 -- 172.21.15.12:0/746290053 shutdown_connections 2024-04-15T15:22:27.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.983+0000 7fdb18e74640 1 -- 172.21.15.12:0/746290053 wait complete. 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.984+0000 7fdb18e74640 1 Processor -- start 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.984+0000 7fdb18e74640 1 -- start start 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb18e74640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb14071240 0x7fdb14154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb18e74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb14071d90 0x7fdb1409e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb18e74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb140ce8e0 0x7fdb1409e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb18e74640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb140704b0 con 0x7fdb140ce8e0 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb18e74640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdb140701d0 con 0x7fdb14071240 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb18e74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fdb14071970 con 0x7fdb14071d90 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb140ce8e0 0x7fdb1409e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb140ce8e0 0x7fdb1409e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42458/0 (socket says 172.21.15.12:42458) 2024-04-15T15:22:27.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 -- 172.21.15.12:0/3256645003 learned_addr learned my addr 172.21.15.12:0/3256645003 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb0bfff640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb14071d90 0x7fdb1409e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb1259c640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb14071240 0x7fdb14154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 -- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb14071d90 msgr2=0x7fdb1409e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb14071d90 0x7fdb1409e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 -- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb14071240 msgr2=0x7fdb14154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb14071240 0x7fdb14154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 -- 172.21.15.12:0/3256645003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1415ee10 con 0x7fdb140ce8e0 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb0bfff640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb14071d90 0x7fdb1409e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.985+0000 7fdb12d9d640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb140ce8e0 0x7fdb1409e780 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00ef00 tx=0x7fdb0c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.986+0000 7fdb0affd640 1 -- 172.21.15.12:0/3256645003 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c018020 con 0x7fdb140ce8e0 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.986+0000 7fdb0affd640 1 -- 172.21.15.12:0/3256645003 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0c00bd20 con 0x7fdb140ce8e0 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.986+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb140cc5b0 con 0x7fdb140ce8e0 2024-04-15T15:22:27.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.986+0000 7fdb0affd640 1 -- 172.21.15.12:0/3256645003 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c014640 con 0x7fdb140ce8e0 2024-04-15T15:22:27.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.986+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb140cc2d0 con 0x7fdb140ce8e0 2024-04-15T15:22:27.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.987+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb140689e0 con 0x7fdb140ce8e0 2024-04-15T15:22:27.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.987+0000 7fdb0affd640 1 -- 172.21.15.12:0/3256645003 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb0c005d10 con 0x7fdb140ce8e0 2024-04-15T15:22:27.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.988+0000 7fdb0affd640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaf406c8f0 0x7fdaf406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:27.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.988+0000 7fdb0affd640 1 -- 172.21.15.12:0/3256645003 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fdb0c08bf50 con 0x7fdb140ce8e0 2024-04-15T15:22:27.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.988+0000 7fdb1259c640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaf406c8f0 0x7fdaf406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:27.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.989+0000 7fdb1259c640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaf406c8f0 0x7fdaf406edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009970 tx=0x7fdb04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:27.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:27.992+0000 7fdb0affd640 1 -- 172.21.15.12:0/3256645003 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb0c022070 con 0x7fdb140ce8e0 2024-04-15T15:22:28.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.253+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fdb140b83b0 con 0x7fdb140ce8e0 2024-04-15T15:22:28.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.255+0000 7fdb0affd640 1 -- 172.21.15.12:0/3256645003 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fdb1415ee10 con 0x7fdb140ce8e0 2024-04-15T15:22:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaf406c8f0 msgr2=0x7fdaf406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaf406c8f0 0x7fdaf406edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009970 tx=0x7fdb04008040 comp rx=0 tx=0).stop 2024-04-15T15:22:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb140ce8e0 msgr2=0x7fdb1409e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb140ce8e0 0x7fdb1409e780 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00ef00 tx=0x7fdb0c00b480 comp rx=0 tx=0).stop 2024-04-15T15:22:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 shutdown_connections 2024-04-15T15:22:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb140ce8e0 0x7fdb1409e780 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb14071d90 0x7fdb1409e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdaf406c8f0 0x7fdaf406edb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 --2- 172.21.15.12:0/3256645003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb14071240 0x7fdb14154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 >> 172.21.15.12:0/3256645003 conn(0x7fdb140b3f40 msgr2=0x7fdb140afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 shutdown_connections 2024-04-15T15:22:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.257+0000 7fdb18e74640 1 -- 172.21.15.12:0/3256645003 wait complete. 2024-04-15T15:22:28.271 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-15T15:22:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.373+0000 7fb7800ce640 1 Processor -- start 2024-04-15T15:22:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.374+0000 7fb7800ce640 1 -- start start 2024-04-15T15:22:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.374+0000 7fb7800ce640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7780701f0 0x7fb778160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.374+0000 7fb7800ce640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 0x7fb7780d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.374+0000 7fb7800ce640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb778070720 con 0x7fb77809a9d0 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.374+0000 7fb7800ce640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb77809ba90 con 0x7fb7780701f0 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.374+0000 7fb7800ce640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb7780d08c0 con 0x7fb77815fb30 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.375+0000 7fb77d669640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7780701f0 0x7fb778160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.375+0000 7fb77d669640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7780701f0 0x7fb778160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42462/0 (socket says 172.21.15.12:42462) 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.375+0000 7fb77d669640 1 -- 172.21.15.12:0/406466141 learned_addr learned my addr 172.21.15.12:0/406466141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.375+0000 7fb77e66b640 1 --2- 172.21.15.12:0/406466141 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 0x7fb7780d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.375+0000 7fb77d669640 1 -- 172.21.15.12:0/406466141 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 msgr2=0x7fb7780d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.375+0000 7fb77d669640 1 --2- 172.21.15.12:0/406466141 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 0x7fb7780d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.375+0000 7fb77d669640 1 -- 172.21.15.12:0/406466141 >> v1:172.21.15.55:6789/0 conn(0x7fb77809a9d0 legacy=0x7fb77809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.375+0000 7fb77d669640 1 -- 172.21.15.12:0/406466141 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb778070a90 con 0x7fb7780701f0 2024-04-15T15:22:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb77d669640 1 --2- 172.21.15.12:0/406466141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7780701f0 0x7fb778160420 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb768009870 tx=0x7fb76802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b9c166bfaa0626 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb77ce68640 1 -- 172.21.15.12:0/406466141 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768034070 con 0x7fb7780701f0 2024-04-15T15:22:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb77ce68640 1 -- 172.21.15.12:0/406466141 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb768030840 con 0x7fb7780701f0 2024-04-15T15:22:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb77ce68640 1 -- 172.21.15.12:0/406466141 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768030c10 con 0x7fb7780701f0 2024-04-15T15:22:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb7800ce640 1 -- 172.21.15.12:0/406466141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7780701f0 msgr2=0x7fb778160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb7800ce640 1 --2- 172.21.15.12:0/406466141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7780701f0 0x7fb778160420 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb768009870 tx=0x7fb76802b980 comp rx=0 tx=0).stop 2024-04-15T15:22:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb7800ce640 1 -- 172.21.15.12:0/406466141 shutdown_connections 2024-04-15T15:22:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb7800ce640 1 --2- 172.21.15.12:0/406466141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7780701f0 0x7fb778160420 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb7800ce640 1 --2- 172.21.15.12:0/406466141 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 0x7fb7780d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.376+0000 7fb7800ce640 1 -- 172.21.15.12:0/406466141 >> 172.21.15.12:0/406466141 conn(0x7fb7780d2460 msgr2=0x7fb7780d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.377+0000 7fb7800ce640 1 -- 172.21.15.12:0/406466141 shutdown_connections 2024-04-15T15:22:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.377+0000 7fb7800ce640 1 -- 172.21.15.12:0/406466141 wait complete. 2024-04-15T15:22:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.378+0000 7fb7800ce640 1 Processor -- start 2024-04-15T15:22:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.378+0000 7fb7800ce640 1 -- start start 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.378+0000 7fb7800ce640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb7780701f0 0x7fb7780afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.378+0000 7fb7800ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb77809a9d0 0x7fb7780aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.378+0000 7fb7800ce640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 0x7fb7780c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.378+0000 7fb7800ce640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb77815ffc0 con 0x7fb77809a9d0 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb7800ce640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb778071050 con 0x7fb7780701f0 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb7800ce640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb7780b1c60 con 0x7fb77815fb30 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb77809a9d0 0x7fb7780aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb77809a9d0 0x7fb7780aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42464/0 (socket says 172.21.15.12:42464) 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 -- 172.21.15.12:0/1403703118 learned_addr learned my addr 172.21.15.12:0/1403703118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77de6a640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb7780701f0 0x7fb7780afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77e66b640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 0x7fb7780c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 -- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 msgr2=0x7fb7780c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 0x7fb7780c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 -- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb7780701f0 msgr2=0x7fb7780afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb7780701f0 0x7fb7780afa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 -- 172.21.15.12:0/1403703118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7780b06b0 con 0x7fb77809a9d0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb77d669640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb77809a9d0 0x7fb7780aff50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb7680097c0 tx=0x7fb768004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb76e7fc640 1 -- 172.21.15.12:0/1403703118 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768048070 con 0x7fb77809a9d0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb76e7fc640 1 -- 172.21.15.12:0/1403703118 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb768030640 con 0x7fb77809a9d0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb76e7fc640 1 -- 172.21.15.12:0/1403703118 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768034040 con 0x7fb77809a9d0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7780b0990 con 0x7fb77809a9d0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.379+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7780b5310 con 0x7fb77809a9d0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.380+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7780689e0 con 0x7fb77809a9d0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.381+0000 7fb76e7fc640 1 -- 172.21.15.12:0/1403703118 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb76804b070 con 0x7fb77809a9d0 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.381+0000 7fb76e7fc640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb75c06c8f0 0x7fb75c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.381+0000 7fb76e7fc640 1 -- 172.21.15.12:0/1403703118 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fb7680ad7a0 con 0x7fb77809a9d0 2024-04-15T15:22:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.382+0000 7fb77de6a640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb75c06c8f0 0x7fb75c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.382+0000 7fb77de6a640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb75c06c8f0 0x7fb75c06edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb774001780 tx=0x7fb774009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.386+0000 7fb76e7fc640 1 -- 172.21.15.12:0/1403703118 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb76807b7d0 con 0x7fb77809a9d0 2024-04-15T15:22:28.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.646+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7fb7780b83e0 con 0x7fb77809a9d0 2024-04-15T15:22:28.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.648+0000 7fb76e7fc640 1 -- 172.21.15.12:0/1403703118 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7fb76807fad0 con 0x7fb77809a9d0 2024-04-15T15:22:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.650+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb75c06c8f0 msgr2=0x7fb75c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.650+0000 7fb7800ce640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb75c06c8f0 0x7fb75c06edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb774001780 tx=0x7fb774009040 comp rx=0 tx=0).stop 2024-04-15T15:22:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.650+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb77809a9d0 msgr2=0x7fb7780aff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.650+0000 7fb7800ce640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb77809a9d0 0x7fb7780aff50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb7680097c0 tx=0x7fb768004730 comp rx=0 tx=0).stop 2024-04-15T15:22:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.651+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 shutdown_connections 2024-04-15T15:22:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.651+0000 7fb7800ce640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb77815fb30 0x7fb7780c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.651+0000 7fb7800ce640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb77809a9d0 0x7fb7780aff50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.651+0000 7fb7800ce640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb75c06c8f0 0x7fb75c06edb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.651+0000 7fb7800ce640 1 --2- 172.21.15.12:0/1403703118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb7780701f0 0x7fb7780afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.651+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 >> 172.21.15.12:0/1403703118 conn(0x7fb7780d2460 msgr2=0x7fb7781601d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:28.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.651+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 shutdown_connections 2024-04-15T15:22:28.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.651+0000 7fb7800ce640 1 -- 172.21.15.12:0/1403703118 wait complete. 2024-04-15T15:22:28.664 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-15T15:22:28.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.767+0000 7f63ef304640 1 Processor -- start 2024-04-15T15:22:28.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.767+0000 7f63ef304640 1 -- start start 2024-04-15T15:22:28.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ef304640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f63e8070940 0x7f63e809b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ef304640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8160310 0x7f63e8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ef304640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63e80d05e0 con 0x7f63e80d0da0 2024-04-15T15:22:28.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ef304640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f63e8071260 con 0x7f63e8070940 2024-04-15T15:22:28.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ef304640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f63e80d1940 con 0x7f63e8160310 2024-04-15T15:22:28.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ed0a0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f63e8070940 0x7f63e809b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ec89f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8160310 0x7f63e8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ed0a0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f63e8070940 0x7f63e809b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:51412/0 (socket says 172.21.15.12:51412) 2024-04-15T15:22:28.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ed8a1640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f63e80d0da0 0x7f63e80d1230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:58054/0 (socket says 172.21.15.12:58054) 2024-04-15T15:22:28.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ec89f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8160310 0x7f63e8070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:42474/0 (socket says 172.21.15.12:42474) 2024-04-15T15:22:28.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.768+0000 7f63ed0a0640 1 -- 172.21.15.12:0/1673480947 learned_addr learned my addr 172.21.15.12:0/1673480947 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:28.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.769+0000 7f63ec89f640 1 -- 172.21.15.12:0/1673480947 >> v2:172.21.15.55:3300/0 conn(0x7f63e8070940 msgr2=0x7f63e809b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:28.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.769+0000 7f63ec89f640 1 --2- 172.21.15.12:0/1673480947 >> v2:172.21.15.55:3300/0 conn(0x7f63e8070940 0x7f63e809b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.769+0000 7f63ec89f640 1 -- 172.21.15.12:0/1673480947 >> v1:172.21.15.55:6789/0 conn(0x7f63e80d0da0 legacy=0x7f63e80d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.769+0000 7f63ec89f640 1 -- 172.21.15.12:0/1673480947 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e815f720 con 0x7f63e8160310 2024-04-15T15:22:28.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ec89f640 1 --2- 172.21.15.12:0/1673480947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8160310 0x7f63e8070d20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63d80097a0 tx=0x7f63d802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=637f95cf8c8c2ac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:28.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63dffff640 1 -- 172.21.15.12:0/1673480947 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8035020 con 0x7f63e8160310 2024-04-15T15:22:28.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63dffff640 1 -- 172.21.15.12:0/1673480947 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d8031830 con 0x7f63e8160310 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63dffff640 1 -- 172.21.15.12:0/1673480947 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8031c00 con 0x7f63e8160310 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ef304640 1 -- 172.21.15.12:0/1673480947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8160310 msgr2=0x7f63e8070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ef304640 1 --2- 172.21.15.12:0/1673480947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8160310 0x7f63e8070d20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63d80097a0 tx=0x7f63d802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ef304640 1 -- 172.21.15.12:0/1673480947 shutdown_connections 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ef304640 1 --2- 172.21.15.12:0/1673480947 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8160310 0x7f63e8070d20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ef304640 1 --2- 172.21.15.12:0/1673480947 >> v2:172.21.15.55:3300/0 conn(0x7f63e8070940 0x7f63e809b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ef304640 1 -- 172.21.15.12:0/1673480947 >> 172.21.15.12:0/1673480947 conn(0x7f63e80b3f40 msgr2=0x7f63e80b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ef304640 1 -- 172.21.15.12:0/1673480947 shutdown_connections 2024-04-15T15:22:28.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.770+0000 7f63ef304640 1 -- 172.21.15.12:0/1673480947 wait complete. 2024-04-15T15:22:28.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ef304640 1 Processor -- start 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ef304640 1 -- start start 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ef304640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8070940 0x7f63e80c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ef304640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63e80d0da0 0x7f63e815a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ef304640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e8160310 0x7f63e80b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ef304640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f63e80d0880 con 0x7f63e80d0da0 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ef304640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f63e815fce0 con 0x7f63e8160310 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ef304640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f63e8160770 con 0x7f63e8070940 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.772+0000 7f63ec89f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63e80d0da0 0x7f63e815a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ec89f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63e80d0da0 0x7f63e815a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42476/0 (socket says 172.21.15.12:42476) 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ec89f640 1 -- 172.21.15.12:0/3460782413 learned_addr learned my addr 172.21.15.12:0/3460782413 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ed0a0640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8070940 0x7f63e80c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ed8a1640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e8160310 0x7f63e80b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ec89f640 1 -- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8070940 msgr2=0x7f63e80c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ec89f640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8070940 0x7f63e80c89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ec89f640 1 -- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e8160310 msgr2=0x7f63e80b73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ec89f640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e8160310 0x7f63e80b73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ec89f640 1 -- 172.21.15.12:0/3460782413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e815d800 con 0x7f63e80d0da0 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ed0a0640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8070940 0x7f63e80c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ed8a1640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e8160310 0x7f63e80b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ec89f640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63e80d0da0 0x7f63e815a990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f63d802bee0 tx=0x7f63d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ddffb640 1 -- 172.21.15.12:0/3460782413 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8035040 con 0x7f63e80d0da0 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ddffb640 1 -- 172.21.15.12:0/3460782413 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d803b920 con 0x7f63e80d0da0 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e815dae0 con 0x7f63e80d0da0 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ddffb640 1 -- 172.21.15.12:0/3460782413 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d803bd10 con 0x7f63e80d0da0 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.773+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e80b78e0 con 0x7f63e80d0da0 2024-04-15T15:22:28.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.774+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63e80b6cb0 con 0x7f63e80d0da0 2024-04-15T15:22:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.775+0000 7f63ddffb640 1 -- 172.21.15.12:0/3460782413 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f63d8038020 con 0x7f63e80d0da0 2024-04-15T15:22:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.775+0000 7f63ddffb640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f63cc06c8f0 0x7f63cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.775+0000 7f63ddffb640 1 -- 172.21.15.12:0/3460782413 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f63d80adc00 con 0x7f63e80d0da0 2024-04-15T15:22:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.776+0000 7f63ed0a0640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f63cc06c8f0 0x7f63cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.776+0000 7f63ed0a0640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f63cc06c8f0 0x7f63cc06edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009a20 tx=0x7f63e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:28.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:28.780+0000 7f63ddffb640 1 -- 172.21.15.12:0/3460782413 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f63d807c160 con 0x7f63e80d0da0 2024-04-15T15:22:29.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.035+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f63e80689e0 con 0x7f63e80d0da0 2024-04-15T15:22:29.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.037+0000 7f63ddffb640 1 -- 172.21.15.12:0/3460782413 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f63d8031800 con 0x7f63e80d0da0 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f63cc06c8f0 msgr2=0x7f63cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f63cc06c8f0 0x7f63cc06edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009a20 tx=0x7f63e4008040 comp rx=0 tx=0).stop 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63e80d0da0 msgr2=0x7f63e815a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63e80d0da0 0x7f63e815a990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f63d802bee0 tx=0x7f63d80046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 shutdown_connections 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f63cc06c8f0 0x7f63cc06edb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e8160310 0x7f63e80b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63e80d0da0 0x7f63e815a990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 --2- 172.21.15.12:0/3460782413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f63e8070940 0x7f63e80c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 >> 172.21.15.12:0/3460782413 conn(0x7f63e80b3f40 msgr2=0x7f63e81609d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:29.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 shutdown_connections 2024-04-15T15:22:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.039+0000 7f63ef304640 1 -- 172.21.15.12:0/3460782413 wait complete. 2024-04-15T15:22:29.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-15T15:22:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.156+0000 7fac1e4c8640 1 Processor -- start 2024-04-15T15:22:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.156+0000 7fac1e4c8640 1 -- start start 2024-04-15T15:22:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.156+0000 7fac1e4c8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac1809b880 0x7fac18070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.156+0000 7fac1e4c8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac180d05e0 con 0x7fac18070940 2024-04-15T15:22:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.156+0000 7fac1e4c8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac18071260 con 0x7fac180d0da0 2024-04-15T15:22:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.156+0000 7fac1e4c8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fac1809bce0 con 0x7fac1809b880 2024-04-15T15:22:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.157+0000 7fac177fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac1809b880 0x7fac18070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.157+0000 7fac177fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac1809b880 0x7fac18070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42490/0 (socket says 172.21.15.12:42490) 2024-04-15T15:22:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.157+0000 7fac177fe640 1 -- 172.21.15.12:0/2454227913 learned_addr learned my addr 172.21.15.12:0/2454227913 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.157+0000 7fac177fe640 1 -- 172.21.15.12:0/2454227913 >> v1:172.21.15.55:6789/0 conn(0x7fac180d0da0 legacy=0x7fac180d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.157+0000 7fac177fe640 1 -- 172.21.15.12:0/2454227913 >> v1:172.21.15.12:6789/0 conn(0x7fac18070940 legacy=0x7fac18160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.157+0000 7fac177fe640 1 -- 172.21.15.12:0/2454227913 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1815f720 con 0x7fac1809b880 2024-04-15T15:22:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac177fe640 1 --2- 172.21.15.12:0/2454227913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1809b880 0x7fac18070d20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009870 tx=0x7fac0c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7b5e6f92fe00d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac16ffd640 1 -- 172.21.15.12:0/2454227913 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c02f840 con 0x7fac1809b880 2024-04-15T15:22:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac16ffd640 1 -- 172.21.15.12:0/2454227913 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac0c02f9e0 con 0x7fac1809b880 2024-04-15T15:22:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac16ffd640 1 -- 172.21.15.12:0/2454227913 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c02fdb0 con 0x7fac1809b880 2024-04-15T15:22:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac1e4c8640 1 -- 172.21.15.12:0/2454227913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1809b880 msgr2=0x7fac18070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac1e4c8640 1 --2- 172.21.15.12:0/2454227913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1809b880 0x7fac18070d20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009870 tx=0x7fac0c02ba20 comp rx=0 tx=0).stop 2024-04-15T15:22:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac1e4c8640 1 -- 172.21.15.12:0/2454227913 shutdown_connections 2024-04-15T15:22:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac1e4c8640 1 --2- 172.21.15.12:0/2454227913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac1809b880 0x7fac18070d20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac1e4c8640 1 -- 172.21.15.12:0/2454227913 >> 172.21.15.12:0/2454227913 conn(0x7fac180b3f40 msgr2=0x7fac180b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac1e4c8640 1 -- 172.21.15.12:0/2454227913 shutdown_connections 2024-04-15T15:22:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.158+0000 7fac1e4c8640 1 -- 172.21.15.12:0/2454227913 wait complete. 2024-04-15T15:22:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.160+0000 7fac1e4c8640 1 Processor -- start 2024-04-15T15:22:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.160+0000 7fac1e4c8640 1 -- start start 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.160+0000 7fac1e4c8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac18070940 0x7fac180c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.160+0000 7fac1e4c8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac1809b880 0x7fac180b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.160+0000 7fac1e4c8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac180d0da0 0x7fac181f6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.160+0000 7fac1e4c8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fac180d0880 con 0x7fac18070940 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.160+0000 7fac1e4c8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fac1815fce0 con 0x7fac1809b880 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.160+0000 7fac1e4c8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fac180d18e0 con 0x7fac180d0da0 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac18070940 0x7fac180c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac18070940 0x7fac180c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42504/0 (socket says 172.21.15.12:42504) 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 -- 172.21.15.12:0/1306376897 learned_addr learned my addr 172.21.15.12:0/1306376897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac177fe640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac1809b880 0x7fac180b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 -- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac180d0da0 msgr2=0x7fac181f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac1ca65640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac180d0da0 0x7fac181f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac180d0da0 0x7fac181f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 -- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac1809b880 msgr2=0x7fac180b7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac1809b880 0x7fac180b7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 -- 172.21.15.12:0/1306376897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac180c82b0 con 0x7fac18070940 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac1ca65640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac180d0da0 0x7fac181f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac177fe640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac1809b880 0x7fac180b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac17fff640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac18070940 0x7fac180c7d70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c400 tx=0x7fac0800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac14ff9640 1 -- 172.21.15.12:0/1306376897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08017070 con 0x7fac18070940 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac14ff9640 1 -- 172.21.15.12:0/1306376897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac080052d0 con 0x7fac18070940 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac180b57a0 con 0x7fac18070940 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac14ff9640 1 -- 172.21.15.12:0/1306376897 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08003830 con 0x7fac18070940 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.161+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac181f50c0 con 0x7fac18070940 2024-04-15T15:22:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.162+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac1806f750 con 0x7fac18070940 2024-04-15T15:22:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.163+0000 7fac14ff9640 1 -- 172.21.15.12:0/1306376897 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac0801a070 con 0x7fac18070940 2024-04-15T15:22:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.163+0000 7fac14ff9640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac0406c8f0 0x7fac0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.163+0000 7fac14ff9640 1 -- 172.21.15.12:0/1306376897 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fac0808abf0 con 0x7fac18070940 2024-04-15T15:22:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.163+0000 7fac177fe640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac0406c8f0 0x7fac0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.164+0000 7fac177fe640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac0406c8f0 0x7fac0406edb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fac0c036000 tx=0x7fac0c039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.168+0000 7fac14ff9640 1 -- 172.21.15.12:0/1306376897 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac1806f750 con 0x7fac18070940 2024-04-15T15:22:29.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.425+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fac180689e0 con 0x7fac18070940 2024-04-15T15:22:29.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.427+0000 7fac14ff9640 1 -- 172.21.15.12:0/1306376897 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7fac08059270 con 0x7fac18070940 2024-04-15T15:22:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac0406c8f0 msgr2=0x7fac0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac0406c8f0 0x7fac0406edb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fac0c036000 tx=0x7fac0c039040 comp rx=0 tx=0).stop 2024-04-15T15:22:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac18070940 msgr2=0x7fac180c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac18070940 0x7fac180c7d70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c400 tx=0x7fac0800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 shutdown_connections 2024-04-15T15:22:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac180d0da0 0x7fac181f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac0406c8f0 0x7fac0406edb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac1809b880 0x7fac180b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 --2- 172.21.15.12:0/1306376897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac18070940 0x7fac180c7d70 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 >> 172.21.15.12:0/1306376897 conn(0x7fac180b3f40 msgr2=0x7fac180b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 shutdown_connections 2024-04-15T15:22:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.429+0000 7fac1e4c8640 1 -- 172.21.15.12:0/1306376897 wait complete. 2024-04-15T15:22:29.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-15T15:22:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.546+0000 7f5ccb07e640 1 Processor -- start 2024-04-15T15:22:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.546+0000 7f5ccb07e640 1 -- start start 2024-04-15T15:22:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5ccb07e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5cc415e790 0x7f5cc40d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5ccb07e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5cc40cfd20 0x7f5cc40d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5ccb07e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 0x7f5cc40d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5ccb07e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc40d2db0 con 0x7f5cc415e790 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5ccb07e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc40d2480 con 0x7f5cc40cfd20 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5ccb07e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc40d2760 con 0x7f5cc40d0f00 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc8e1a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5cc415e790 0x7f5cc40d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc8e1a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5cc415e790 0x7f5cc40d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42510/0 (socket says 172.21.15.12:42510) 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc8e1a640 1 -- 172.21.15.12:0/3320243569 learned_addr learned my addr 172.21.15.12:0/3320243569 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc961b640 1 --2- 172.21.15.12:0/3320243569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 0x7f5cc40d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc3fff640 1 --2- 172.21.15.12:0/3320243569 >> v2:172.21.15.55:3300/0 conn(0x7f5cc40cfd20 0x7f5cc40d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc8e1a640 1 -- 172.21.15.12:0/3320243569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 msgr2=0x7f5cc40d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc8e1a640 1 --2- 172.21.15.12:0/3320243569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 0x7f5cc40d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc8e1a640 1 -- 172.21.15.12:0/3320243569 >> v2:172.21.15.55:3300/0 conn(0x7f5cc40cfd20 msgr2=0x7f5cc40d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc8e1a640 1 --2- 172.21.15.12:0/3320243569 >> v2:172.21.15.55:3300/0 conn(0x7f5cc40cfd20 0x7f5cc40d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.547+0000 7f5cc8e1a640 1 -- 172.21.15.12:0/3320243569 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc40d2a40 con 0x7f5cc415e790 2024-04-15T15:22:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.548+0000 7f5cc8e1a640 1 --2- 172.21.15.12:0/3320243569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40d3240 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5cc408e510 tx=0x7f5cb802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d670ea5fc4af6dc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.548+0000 7f5cc37fe640 1 -- 172.21.15.12:0/3320243569 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8035020 con 0x7f5cc415e790 2024-04-15T15:22:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.548+0000 7f5cc37fe640 1 -- 172.21.15.12:0/3320243569 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb8031870 con 0x7f5cc415e790 2024-04-15T15:22:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.548+0000 7f5cc37fe640 1 -- 172.21.15.12:0/3320243569 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8031c40 con 0x7f5cc415e790 2024-04-15T15:22:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 -- 172.21.15.12:0/3320243569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 msgr2=0x7f5cc40d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/3320243569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40d3240 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5cc408e510 tx=0x7f5cb802b910 comp rx=0 tx=0).stop 2024-04-15T15:22:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 -- 172.21.15.12:0/3320243569 shutdown_connections 2024-04-15T15:22:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/3320243569 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 0x7f5cc40d13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/3320243569 >> v2:172.21.15.55:3300/0 conn(0x7f5cc40cfd20 0x7f5cc40d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/3320243569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40d3240 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 -- 172.21.15.12:0/3320243569 >> 172.21.15.12:0/3320243569 conn(0x7f5cc408bf60 msgr2=0x7f5cc40d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 -- 172.21.15.12:0/3320243569 shutdown_connections 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.549+0000 7f5ccb07e640 1 -- 172.21.15.12:0/3320243569 wait complete. 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.550+0000 7f5ccb07e640 1 Processor -- start 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.550+0000 7f5ccb07e640 1 -- start start 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5ccb07e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cc40cfd20 0x7f5cc409e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5ccb07e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 0x7f5cc40cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5ccb07e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5ccb07e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc415ecf0 con 0x7f5cc415e790 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5ccb07e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc415eb70 con 0x7f5cc40cfd20 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5ccb07e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc40b74f0 con 0x7f5cc40d0f00 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40a1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42526/0 (socket says 172.21.15.12:42526) 2024-04-15T15:22:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 -- 172.21.15.12:0/2548204320 learned_addr learned my addr 172.21.15.12:0/2548204320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc8e1a640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cc40cfd20 0x7f5cc409e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc3fff640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 0x7f5cc40cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 -- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 msgr2=0x7f5cc40cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 0x7f5cc40cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 -- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cc40cfd20 msgr2=0x7f5cc409e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cc40cfd20 0x7f5cc409e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 -- 172.21.15.12:0/2548204320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc40b3910 con 0x7f5cc415e790 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc8e1a640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cc40cfd20 0x7f5cc409e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc961b640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40a1ca0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc007e40 tx=0x7f5cbc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc17fa640 1 -- 172.21.15.12:0/2548204320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc014620 con 0x7f5cc415e790 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.551+0000 7f5cc17fa640 1 -- 172.21.15.12:0/2548204320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cbc019070 con 0x7f5cc415e790 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.552+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc40b3bf0 con 0x7f5cc415e790 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.552+0000 7f5cc17fa640 1 -- 172.21.15.12:0/2548204320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc014a20 con 0x7f5cc415e790 2024-04-15T15:22:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.552+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc40b41b0 con 0x7f5cc415e790 2024-04-15T15:22:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.553+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cc40689e0 con 0x7f5cc415e790 2024-04-15T15:22:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.553+0000 7f5cc17fa640 1 -- 172.21.15.12:0/2548204320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5cbc014bc0 con 0x7f5cc415e790 2024-04-15T15:22:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.553+0000 7f5cc17fa640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ca806c8f0 0x7f5ca806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.554+0000 7f5cc17fa640 1 -- 172.21.15.12:0/2548204320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f5cbc08c2a0 con 0x7f5cc415e790 2024-04-15T15:22:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.554+0000 7f5cc8e1a640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ca806c8f0 0x7f5ca806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.554+0000 7f5cc8e1a640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ca806c8f0 0x7f5ca806edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8036000 tx=0x7f5cb803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.558+0000 7f5cc17fa640 1 -- 172.21.15.12:0/2548204320 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5cbc05a890 con 0x7f5cc415e790 2024-04-15T15:22:29.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.821+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f5cc40d1ce0 con 0x7f5cc415e790 2024-04-15T15:22:29.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.823+0000 7f5cc17fa640 1 -- 172.21.15.12:0/2548204320 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f5cbc08c0c0 con 0x7f5cc415e790 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ca806c8f0 msgr2=0x7f5ca806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ca806c8f0 0x7f5ca806edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8036000 tx=0x7f5cb803c040 comp rx=0 tx=0).stop 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 msgr2=0x7f5cc40a1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40a1ca0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc007e40 tx=0x7f5cbc00e4b0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 shutdown_connections 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5cc415e790 0x7f5cc40a1ca0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5cc40d0f00 0x7f5cc40cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f5ca806c8f0 0x7f5ca806edb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 --2- 172.21.15.12:0/2548204320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cc40cfd20 0x7f5cc409e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 >> 172.21.15.12:0/2548204320 conn(0x7f5cc408bf60 msgr2=0x7f5cc40b8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 shutdown_connections 2024-04-15T15:22:29.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.825+0000 7f5ccb07e640 1 -- 172.21.15.12:0/2548204320 wait complete. 2024-04-15T15:22:29.839 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:29.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.940+0000 7f97dd075640 1 Processor -- start 2024-04-15T15:22:29.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.940+0000 7f97dd075640 1 -- start start 2024-04-15T15:22:29.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97dd075640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97d815e830 0x7f97d80b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97dd075640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f97d80b53c0 0x7f97d80b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97dd075640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d8067e30 con 0x7f97d815ec10 2024-04-15T15:22:29.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97dd075640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f97d80b3c10 con 0x7f97d815e830 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97dd075640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f97d80b3ef0 con 0x7f97d80b53c0 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97d6d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97d815e830 0x7f97d80b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97d6d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97d815e830 0x7f97d80b4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42532/0 (socket says 172.21.15.12:42532) 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97d6d9d640 1 -- 172.21.15.12:0/3547338267 learned_addr learned my addr 172.21.15.12:0/3547338267 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97d759e640 1 --2- 172.21.15.12:0/3547338267 >> v2:172.21.15.55:3300/0 conn(0x7f97d80b53c0 0x7f97d80b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97d6d9d640 1 -- 172.21.15.12:0/3547338267 >> v2:172.21.15.55:3300/0 conn(0x7f97d80b53c0 msgr2=0x7f97d80b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97d6d9d640 1 --2- 172.21.15.12:0/3547338267 >> v2:172.21.15.55:3300/0 conn(0x7f97d80b53c0 0x7f97d80b5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97d6d9d640 1 -- 172.21.15.12:0/3547338267 >> v1:172.21.15.55:6789/0 conn(0x7f97d815ec10 legacy=0x7f97d8067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.941+0000 7f97d6d9d640 1 -- 172.21.15.12:0/3547338267 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d80b41d0 con 0x7f97d815e830 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.942+0000 7f97d6d9d640 1 --2- 172.21.15.12:0/3547338267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d815e830 0x7f97d80b4cb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f97d80b51f0 tx=0x7f97c802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ac4e4cefe688c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.942+0000 7f97d5d9b640 1 -- 172.21.15.12:0/3547338267 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c802c9f0 con 0x7f97d815e830 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.942+0000 7f97d5d9b640 1 -- 172.21.15.12:0/3547338267 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c802cb90 con 0x7f97d815e830 2024-04-15T15:22:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.942+0000 7f97d5d9b640 1 -- 172.21.15.12:0/3547338267 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c80343f0 con 0x7f97d815e830 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.943+0000 7f97dd075640 1 -- 172.21.15.12:0/3547338267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d815e830 msgr2=0x7f97d80b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.943+0000 7f97dd075640 1 --2- 172.21.15.12:0/3547338267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d815e830 0x7f97d80b4cb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f97d80b51f0 tx=0x7f97c802be50 comp rx=0 tx=0).stop 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.943+0000 7f97dd075640 1 -- 172.21.15.12:0/3547338267 shutdown_connections 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.943+0000 7f97dd075640 1 --2- 172.21.15.12:0/3547338267 >> v2:172.21.15.55:3300/0 conn(0x7f97d80b53c0 0x7f97d80b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.943+0000 7f97dd075640 1 --2- 172.21.15.12:0/3547338267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d815e830 0x7f97d80b4cb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.943+0000 7f97dd075640 1 -- 172.21.15.12:0/3547338267 >> 172.21.15.12:0/3547338267 conn(0x7f97d8073360 msgr2=0x7f97d806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.943+0000 7f97dd075640 1 -- 172.21.15.12:0/3547338267 shutdown_connections 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.943+0000 7f97dd075640 1 -- 172.21.15.12:0/3547338267 wait complete. 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.944+0000 7f97dd075640 1 Processor -- start 2024-04-15T15:22:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.944+0000 7f97dd075640 1 -- start start 2024-04-15T15:22:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97dd075640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97d80b53c0 0x7f97d81e7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97dd075640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f97d815e830 0x7f97d81596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97dd075640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d80c62b0 0x7f97d815cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97dd075640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f97d815ff60 con 0x7f97d80c62b0 2024-04-15T15:22:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97dd075640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f97d80b4790 con 0x7f97d80b53c0 2024-04-15T15:22:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97dd075640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f97d81600e0 con 0x7f97d815e830 2024-04-15T15:22:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d80c62b0 0x7f97d815cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d80c62b0 0x7f97d815cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42540/0 (socket says 172.21.15.12:42540) 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d759e640 1 -- 172.21.15.12:0/1751535600 learned_addr learned my addr 172.21.15.12:0/1751535600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d6d9d640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97d80b53c0 0x7f97d81e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d659c640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f97d815e830 0x7f97d81596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d759e640 1 -- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f97d815e830 msgr2=0x7f97d81596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d759e640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f97d815e830 0x7f97d81596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d759e640 1 -- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97d80b53c0 msgr2=0x7f97d81e7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d759e640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97d80b53c0 0x7f97d81e7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.945+0000 7f97d759e640 1 -- 172.21.15.12:0/1751535600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d8159c10 con 0x7f97d80c62b0 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.946+0000 7f97d759e640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d80c62b0 0x7f97d815cf40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c7f0 tx=0x7f97d000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.946+0000 7f97cf7fe640 1 -- 172.21.15.12:0/1751535600 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0015070 con 0x7f97d80c62b0 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.946+0000 7f97cf7fe640 1 -- 172.21.15.12:0/1751535600 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d0004220 con 0x7f97d80c62b0 2024-04-15T15:22:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.946+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d81e8830 con 0x7f97d80c62b0 2024-04-15T15:22:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.946+0000 7f97cf7fe640 1 -- 172.21.15.12:0/1751535600 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0007620 con 0x7f97d80c62b0 2024-04-15T15:22:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.946+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d81e7f00 con 0x7f97d80c62b0 2024-04-15T15:22:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.947+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d815e3e0 con 0x7f97d80c62b0 2024-04-15T15:22:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.947+0000 7f97cf7fe640 1 -- 172.21.15.12:0/1751535600 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f97d00077c0 con 0x7f97d80c62b0 2024-04-15T15:22:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.948+0000 7f97cf7fe640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f97b806c8f0 0x7f97b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.948+0000 7f97cf7fe640 1 -- 172.21.15.12:0/1751535600 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f97d008bb30 con 0x7f97d80c62b0 2024-04-15T15:22:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.948+0000 7f97d6d9d640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f97b806c8f0 0x7f97b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.948+0000 7f97d6d9d640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f97b806c8f0 0x7f97b806edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f97c8035000 tx=0x7f97c8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:29.952+0000 7f97cf7fe640 1 -- 172.21.15.12:0/1751535600 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97d815e3e0 con 0x7f97d80c62b0 2024-04-15T15:22:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.137+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97d80689e0 con 0x7f97d80c62b0 2024-04-15T15:22:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.138+0000 7f97cf7fe640 1 -- 172.21.15.12:0/1751535600 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13128 (secure 0 0 0) 0x7f97d0059a50 con 0x7f97d80c62b0 2024-04-15T15:22:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:30.141 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":35,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:22:27.085738+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f97b806c8f0 msgr2=0x7f97b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f97b806c8f0 0x7f97b806edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f97c8035000 tx=0x7f97c8030000 comp rx=0 tx=0).stop 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d80c62b0 msgr2=0x7f97d815cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d80c62b0 0x7f97d815cf40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c7f0 tx=0x7f97d000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 shutdown_connections 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97d80c62b0 0x7f97d815cf40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f97d815e830 0x7f97d81596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f97b806c8f0 0x7f97b806edb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 --2- 172.21.15.12:0/1751535600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97d80b53c0 0x7f97d81e7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 >> 172.21.15.12:0/1751535600 conn(0x7f97d8073360 msgr2=0x7f97d80b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:30.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 shutdown_connections 2024-04-15T15:22:30.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.140+0000 7f97dd075640 1 -- 172.21.15.12:0/1751535600 wait complete. 2024-04-15T15:22:30.154 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:22:30.154 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.259+0000 7fdb5111c640 1 Processor -- start 2024-04-15T15:22:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.259+0000 7fdb5111c640 1 -- start start 2024-04-15T15:22:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb5111c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdb4c15e690 0x7fdb4c0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb5111c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb4c0cbcc0 0x7fdb4c1622b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb5111c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb4c0cc430 con 0x7fdb4c06e3e0 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb5111c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fdb4c0cc150 con 0x7fdb4c0cbcc0 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb5111c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fdb4c06ba80 con 0x7fdb4c15e690 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb4a59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb4c0cbcc0 0x7fdb4c1622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb4a59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb4c0cbcc0 0x7fdb4c1622b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42550/0 (socket says 172.21.15.12:42550) 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb4a59c640 1 -- 172.21.15.12:0/2080767662 learned_addr learned my addr 172.21.15.12:0/2080767662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb4ad9d640 1 --2- 172.21.15.12:0/2080767662 >> v2:172.21.15.55:3300/0 conn(0x7fdb4c15e690 0x7fdb4c0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb4a59c640 1 -- 172.21.15.12:0/2080767662 >> v2:172.21.15.55:3300/0 conn(0x7fdb4c15e690 msgr2=0x7fdb4c0cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb4a59c640 1 --2- 172.21.15.12:0/2080767662 >> v2:172.21.15.55:3300/0 conn(0x7fdb4c15e690 0x7fdb4c0cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb4a59c640 1 -- 172.21.15.12:0/2080767662 >> v1:172.21.15.12:6789/0 conn(0x7fdb4c06e3e0 legacy=0x7fdb4c06c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.260+0000 7fdb4a59c640 1 -- 172.21.15.12:0/2080767662 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4c06bd60 con 0x7fdb4c0cbcc0 2024-04-15T15:22:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.261+0000 7fdb4a59c640 1 --2- 172.21.15.12:0/2080767662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c0cbcc0 0x7fdb4c1622b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c06d580 tx=0x7fdb3402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7dedec9126ffa7c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.261+0000 7fdb49d9b640 1 -- 172.21.15.12:0/2080767662 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34034070 con 0x7fdb4c0cbcc0 2024-04-15T15:22:30.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.261+0000 7fdb49d9b640 1 -- 172.21.15.12:0/2080767662 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb34030860 con 0x7fdb4c0cbcc0 2024-04-15T15:22:30.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.261+0000 7fdb49d9b640 1 -- 172.21.15.12:0/2080767662 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34030c30 con 0x7fdb4c0cbcc0 2024-04-15T15:22:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.262+0000 7fdb5111c640 1 -- 172.21.15.12:0/2080767662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c0cbcc0 msgr2=0x7fdb4c1622b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.262+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2080767662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c0cbcc0 0x7fdb4c1622b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c06d580 tx=0x7fdb3402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.262+0000 7fdb5111c640 1 -- 172.21.15.12:0/2080767662 shutdown_connections 2024-04-15T15:22:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.262+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2080767662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c0cbcc0 0x7fdb4c1622b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.262+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2080767662 >> v2:172.21.15.55:3300/0 conn(0x7fdb4c15e690 0x7fdb4c0cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.262+0000 7fdb5111c640 1 -- 172.21.15.12:0/2080767662 >> 172.21.15.12:0/2080767662 conn(0x7fdb4c0cd6e0 msgr2=0x7fdb4c0cdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:30.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.262+0000 7fdb5111c640 1 -- 172.21.15.12:0/2080767662 shutdown_connections 2024-04-15T15:22:30.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.262+0000 7fdb5111c640 1 -- 172.21.15.12:0/2080767662 wait complete. 2024-04-15T15:22:30.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.263+0000 7fdb5111c640 1 Processor -- start 2024-04-15T15:22:30.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb5111c640 1 -- start start 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb5111c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb4c06e3e0 0x7fdb4c1f4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb5111c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c15e690 0x7fdb4c157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb5111c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb4c1547a0 0x7fdb4c157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb5111c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c0cb7f0 con 0x7fdb4c15e690 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb5111c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c15ea70 con 0x7fdb4c06e3e0 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb5111c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fdb4c0cb970 con 0x7fdb4c1547a0 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb4a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c15e690 0x7fdb4c157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb4a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c15e690 0x7fdb4c157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42552/0 (socket says 172.21.15.12:42552) 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb4a59c640 1 -- 172.21.15.12:0/2319371440 learned_addr learned my addr 172.21.15.12:0/2319371440 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.264+0000 7fdb4ad9d640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb4c06e3e0 0x7fdb4c1f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb4b59e640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb4c1547a0 0x7fdb4c157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb4a59c640 1 -- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb4c1547a0 msgr2=0x7fdb4c157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb4a59c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb4c1547a0 0x7fdb4c157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb4a59c640 1 -- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb4c06e3e0 msgr2=0x7fdb4c1f4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb4a59c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb4c06e3e0 0x7fdb4c1f4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb4a59c640 1 -- 172.21.15.12:0/2319371440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4c157410 con 0x7fdb4c15e690 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb4a59c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c15e690 0x7fdb4c157ab0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdb34000c00 tx=0x7fdb34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb437fe640 1 -- 172.21.15.12:0/2319371440 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34048070 con 0x7fdb4c15e690 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb437fe640 1 -- 172.21.15.12:0/2319371440 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb34030660 con 0x7fdb4c15e690 2024-04-15T15:22:30.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4c1576f0 con 0x7fdb4c15e690 2024-04-15T15:22:30.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb437fe640 1 -- 172.21.15.12:0/2319371440 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34034040 con 0x7fdb4c15e690 2024-04-15T15:22:30.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.265+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb4c1f4940 con 0x7fdb4c15e690 2024-04-15T15:22:30.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.266+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb4c0ccfc0 con 0x7fdb4c15e690 2024-04-15T15:22:30.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.267+0000 7fdb437fe640 1 -- 172.21.15.12:0/2319371440 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb3404b070 con 0x7fdb4c15e690 2024-04-15T15:22:30.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.267+0000 7fdb437fe640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdb2c06c920 0x7fdb2c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.267+0000 7fdb437fe640 1 -- 172.21.15.12:0/2319371440 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fdb340ad870 con 0x7fdb4c15e690 2024-04-15T15:22:30.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.267+0000 7fdb4ad9d640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdb2c06c920 0x7fdb2c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.268+0000 7fdb4ad9d640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdb2c06c920 0x7fdb2c06ede0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009aa0 tx=0x7fdb3c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.272+0000 7fdb437fe640 1 -- 172.21.15.12:0/2319371440 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb4c0ccfc0 con 0x7fdb4c15e690 2024-04-15T15:22:30.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.455+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdb4c0690f0 con 0x7fdb4c15e690 2024-04-15T15:22:30.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.455+0000 7fdb437fe640 1 -- 172.21.15.12:0/2319371440 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7fdb3407b8a0 con 0x7fdb4c15e690 2024-04-15T15:22:30.458 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdb2c06c920 msgr2=0x7fdb2c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdb2c06c920 0x7fdb2c06ede0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009aa0 tx=0x7fdb3c009190 comp rx=0 tx=0).stop 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c15e690 msgr2=0x7fdb4c157ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c15e690 0x7fdb4c157ab0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdb34000c00 tx=0x7fdb34004730 comp rx=0 tx=0).stop 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 shutdown_connections 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdb4c1547a0 0x7fdb4c157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb4c15e690 0x7fdb4c157ab0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdb2c06c920 0x7fdb2c06ede0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 --2- 172.21.15.12:0/2319371440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb4c06e3e0 0x7fdb4c1f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 >> 172.21.15.12:0/2319371440 conn(0x7fdb4c0cd6e0 msgr2=0x7fdb4c0cfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 shutdown_connections 2024-04-15T15:22:30.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.457+0000 7fdb5111c640 1 -- 172.21.15.12:0/2319371440 wait complete. 2024-04-15T15:22:30.471 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-15T15:22:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.573+0000 7f4716e63640 1 Processor -- start 2024-04-15T15:22:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.573+0000 7f4716e63640 1 -- start start 2024-04-15T15:22:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4716e63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471015e700 0x7f47100b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4716e63640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f47100d2360 0x7f47100d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4716e63640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47100b47a0 con 0x7f471015eae0 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4716e63640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f47100b53b0 con 0x7f47100d2360 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4716e63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f47100b4a80 con 0x7f471015e700 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4715400640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f471015eae0 0x7f47100cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60746/0 (socket says 172.21.15.12:60746) 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4715400640 1 -- 172.21.15.12:0/112178061 learned_addr learned my addr 172.21.15.12:0/112178061 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f470ffff640 1 --2- 172.21.15.12:0/112178061 >> v2:172.21.15.55:3300/0 conn(0x7f47100d2360 0x7f47100d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4714bff640 1 --2- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471015e700 0x7f47100b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f470f7fe640 1 -- 172.21.15.12:0/112178061 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364549930 0 0) 0x7f47100b47a0 con 0x7f471015eae0 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f470f7fe640 1 -- 172.21.15.12:0/112178061 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46fc003520 con 0x7f471015eae0 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f470ffff640 1 -- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471015e700 msgr2=0x7f47100b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f470ffff640 1 --2- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471015e700 0x7f47100b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f470f7fe640 1 -- 172.21.15.12:0/112178061 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2682560534 0 0) 0x7f46fc003520 con 0x7f471015eae0 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f470ffff640 1 -- 172.21.15.12:0/112178061 >> v1:172.21.15.12:6789/0 conn(0x7f471015eae0 legacy=0x7f47100cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f470ffff640 1 -- 172.21.15.12:0/112178061 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47100b4d60 con 0x7f47100d2360 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.574+0000 7f4714bff640 1 --2- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471015e700 0x7f47100b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.575+0000 7f470ffff640 1 --2- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47100d2360 0x7f47100d2b60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f47000097a0 tx=0x7f470002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d26f8446e4eff9de server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.575+0000 7f470f7fe640 1 -- 172.21.15.12:0/112178061 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700037070 con 0x7f47100d2360 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.575+0000 7f470f7fe640 1 -- 172.21.15.12:0/112178061 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4700030510 con 0x7f47100d2360 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.575+0000 7f470f7fe640 1 -- 172.21.15.12:0/112178061 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47080019f0 con 0x7f47100d2360 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.576+0000 7f4716e63640 1 -- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47100d2360 msgr2=0x7f47100d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.576+0000 7f4716e63640 1 --2- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47100d2360 0x7f47100d2b60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f47000097a0 tx=0x7f470002b790 comp rx=0 tx=0).stop 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.576+0000 7f4716e63640 1 -- 172.21.15.12:0/112178061 shutdown_connections 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.576+0000 7f4716e63640 1 --2- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47100d2360 0x7f47100d2b60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.576+0000 7f4716e63640 1 --2- 172.21.15.12:0/112178061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f471015e700 0x7f47100b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.576+0000 7f4716e63640 1 -- 172.21.15.12:0/112178061 >> 172.21.15.12:0/112178061 conn(0x7f471008bf60 msgr2=0x7f47100d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.576+0000 7f4716e63640 1 -- 172.21.15.12:0/112178061 shutdown_connections 2024-04-15T15:22:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.576+0000 7f4716e63640 1 -- 172.21.15.12:0/112178061 wait complete. 2024-04-15T15:22:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.577+0000 7f4716e63640 1 Processor -- start 2024-04-15T15:22:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.577+0000 7f4716e63640 1 -- start start 2024-04-15T15:22:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f4716e63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47100d2360 0x7f4710161920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f4716e63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471015e700 0x7f4710153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f4716e63640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f471015eae0 0x7f471015efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f4716e63640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f47100b5d80 con 0x7f471015e700 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f4716e63640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f47100b4450 con 0x7f471015eae0 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f4716e63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f47100d1d40 con 0x7f47100d2360 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f470ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471015e700 0x7f4710153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f470ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471015e700 0x7f4710153d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42564/0 (socket says 172.21.15.12:42564) 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f470ffff640 1 -- 172.21.15.12:0/3036345275 learned_addr learned my addr 172.21.15.12:0/3036345275 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f470ffff640 1 -- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47100d2360 msgr2=0x7f4710161920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.578+0000 7f470ffff640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47100d2360 0x7f4710161920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f470ffff640 1 -- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f471015eae0 msgr2=0x7f471015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f470ffff640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f471015eae0 0x7f471015efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f470ffff640 1 -- 172.21.15.12:0/3036345275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4710154270 con 0x7f471015e700 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f470ffff640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471015e700 0x7f4710153d30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4700000c00 tx=0x7f47000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f470d7fa640 1 -- 172.21.15.12:0/3036345275 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700041020 con 0x7f471015e700 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f470d7fa640 1 -- 172.21.15.12:0/3036345275 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470003c870 con 0x7f471015e700 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47100cc600 con 0x7f471015e700 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f470d7fa640 1 -- 172.21.15.12:0/3036345275 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700037040 con 0x7f471015e700 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.579+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47100c9390 con 0x7f471015e700 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.580+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47100689e0 con 0x7f471015e700 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.580+0000 7f470d7fa640 1 -- 172.21.15.12:0/3036345275 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4700044020 con 0x7f471015e700 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.581+0000 7f470d7fa640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f006c8f0 0x7f46f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.581+0000 7f470d7fa640 1 -- 172.21.15.12:0/3036345275 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f47000ad240 con 0x7f471015e700 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.581+0000 7f4714bff640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f006c8f0 0x7f46f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.582+0000 7f4714bff640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f006c8f0 0x7f46f006edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f46f8006e20 tx=0x7f46f8006d70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.586+0000 7f470d7fa640 1 -- 172.21.15.12:0/3036345275 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f470004b030 con 0x7f471015e700 2024-04-15T15:22:30.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.772+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f47100ba500 con 0x7f471015e700 2024-04-15T15:22:30.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.772+0000 7f470d7fa640 1 -- 172.21.15.12:0/3036345275 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7f470007b8c0 con 0x7f471015e700 2024-04-15T15:22:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:30.775 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-04-15T15:22:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:22:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-04-15T15:22:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-04-15T15:22:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:30.775 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f006c8f0 msgr2=0x7f46f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f006c8f0 0x7f46f006edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f46f8006e20 tx=0x7f46f8006d70 comp rx=0 tx=0).stop 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471015e700 msgr2=0x7f4710153d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471015e700 0x7f4710153d30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4700000c00 tx=0x7f47000046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 shutdown_connections 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f46f006c8f0 0x7f46f006edb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f471015eae0 0x7f471015efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f471015e700 0x7f4710153d30 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 --2- 172.21.15.12:0/3036345275 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f47100d2360 0x7f4710161920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 >> 172.21.15.12:0/3036345275 conn(0x7f471008bf60 msgr2=0x7f471006e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:30.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 shutdown_connections 2024-04-15T15:22:30.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.774+0000 7f4716e63640 1 -- 172.21.15.12:0/3036345275 wait complete. 2024-04-15T15:22:30.787 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-15T15:22:30.788 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-15T15:22:30.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.893+0000 7ff64eb82640 1 Processor -- start 2024-04-15T15:22:30.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.894+0000 7ff64eb82640 1 -- start start 2024-04-15T15:22:30.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.894+0000 7ff64eb82640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff64815e8c0 0x7ff6480d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.894+0000 7ff64eb82640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 0x7ff6480d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.894+0000 7ff64eb82640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6480d2340 con 0x7ff6480d0140 2024-04-15T15:22:30.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.894+0000 7ff64eb82640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff64815eca0 con 0x7ff64815e8c0 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.894+0000 7ff64eb82640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff6480d2620 con 0x7ff6480cfcb0 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.894+0000 7ff64d11f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff6480d0140 0x7ff64806c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60752/0 (socket says 172.21.15.12:60752) 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.894+0000 7ff64d11f640 1 -- 172.21.15.12:0/3412836473 learned_addr learned my addr 172.21.15.12:0/3412836473 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63ffff640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 0x7ff6480d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff64c91e640 1 --2- 172.21.15.12:0/3412836473 >> v2:172.21.15.55:3300/0 conn(0x7ff64815e8c0 0x7ff6480d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3214645277 0 0) 0x7ff6480d2340 con 0x7ff6480d0140 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff630003520 con 0x7ff6480d0140 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1767569407 0 0) 0x7ff630003520 con 0x7ff6480d0140 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 msgr2=0x7ff6480d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 0x7ff6480d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 >> v2:172.21.15.55:3300/0 conn(0x7ff64815e8c0 msgr2=0x7ff6480d3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 --2- 172.21.15.12:0/3412836473 >> v2:172.21.15.55:3300/0 conn(0x7ff64815e8c0 0x7ff6480d3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6480d2900 con 0x7ff6480d0140 2024-04-15T15:22:30.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7ff6400019f0 con 0x7ff6480d0140 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63ffff640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 0x7ff6480d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 >> v1:172.21.15.12:6789/0 conn(0x7ff6480d0140 legacy=0x7ff64806c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6300039b0 0x7ff630003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 0x7ff630004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff630004d40 0x7ff6300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff64815eca0 con 0x7ff630004d40 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6480d2620 con 0x7ff6300039b0 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff63000b9c0 con 0x7ff6480cfcb0 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff64d11f640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff630004d40 0x7ff6300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff64c91e640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6300039b0 0x7ff630003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff63ffff640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 0x7ff630004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.895+0000 7ff64d11f640 1 -- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 msgr2=0x7ff630004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.896+0000 7ff64d11f640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6480cfcb0 0x7ff630004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.896+0000 7ff64d11f640 1 -- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6300039b0 msgr2=0x7ff630003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.896+0000 7ff64d11f640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6300039b0 0x7ff630003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.896+0000 7ff64d11f640 1 -- 172.21.15.12:0/3412836473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff630003520 con 0x7ff630004d40 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.896+0000 7ff64d11f640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff630004d40 0x7ff6300051c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff64000a8a0 tx=0x7ff630005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9574837bd32537e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.896+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64000a4c0 con 0x7ff630004d40 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.896+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff640034650 con 0x7ff630004d40 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.896+0000 7ff63f7fe640 1 -- 172.21.15.12:0/3412836473 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640034a40 con 0x7ff630004d40 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.897+0000 7ff64eb82640 1 -- 172.21.15.12:0/3412836473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff630004d40 msgr2=0x7ff6300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.897+0000 7ff64eb82640 1 --2- 172.21.15.12:0/3412836473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff630004d40 0x7ff6300051c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff64000a8a0 tx=0x7ff630005830 comp rx=0 tx=0).stop 2024-04-15T15:22:30.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.897+0000 7ff64d11f640 1 -- 172.21.15.12:0/3412836473 reap_dead start 2024-04-15T15:22:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.897+0000 7ff64eb82640 1 -- 172.21.15.12:0/3412836473 shutdown_connections 2024-04-15T15:22:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.897+0000 7ff64eb82640 1 -- 172.21.15.12:0/3412836473 >> 172.21.15.12:0/3412836473 conn(0x7ff64808bf60 msgr2=0x7ff6480d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.897+0000 7ff64eb82640 1 -- 172.21.15.12:0/3412836473 shutdown_connections 2024-04-15T15:22:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.897+0000 7ff64eb82640 1 -- 172.21.15.12:0/3412836473 wait complete. 2024-04-15T15:22:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.898+0000 7ff64eb82640 1 Processor -- start 2024-04-15T15:22:30.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.898+0000 7ff64eb82640 1 -- start start 2024-04-15T15:22:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64eb82640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630004d40 0x7ff6480ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64eb82640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6481f7100 0x7ff6480c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64eb82640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64815b7a0 0x7ff64815bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64eb82640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff6480d4f60 con 0x7ff64815b7a0 2024-04-15T15:22:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64eb82640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6480d4de0 con 0x7ff630004d40 2024-04-15T15:22:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64eb82640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff64806d080 con 0x7ff6481f7100 2024-04-15T15:22:30.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64d11f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64815b7a0 0x7ff64815bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64d11f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64815b7a0 0x7ff64815bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42590/0 (socket says 172.21.15.12:42590) 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64d11f640 1 -- 172.21.15.12:0/3774641734 learned_addr learned my addr 172.21.15.12:0/3774641734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff63ffff640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6481f7100 0x7ff6480c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.899+0000 7ff64c91e640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630004d40 0x7ff6480ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff64d11f640 1 -- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6481f7100 msgr2=0x7ff6480c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff64d11f640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6481f7100 0x7ff6480c7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff64d11f640 1 -- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630004d40 msgr2=0x7ff6480ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff64d11f640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630004d40 0x7ff6480ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff64d11f640 1 -- 172.21.15.12:0/3774641734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6481f7580 con 0x7ff64815b7a0 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff63ffff640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6481f7100 0x7ff6480c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff64d11f640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64815b7a0 0x7ff64815bc20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff64000a9c0 tx=0x7ff640001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff63d7fa640 1 -- 172.21.15.12:0/3774641734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640045070 con 0x7ff64815b7a0 2024-04-15T15:22:30.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff63d7fa640 1 -- 172.21.15.12:0/3774641734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64003a040 con 0x7ff64815b7a0 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6480c8180 con 0x7ff64815b7a0 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff63d7fa640 1 -- 172.21.15.12:0/3774641734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640002010 con 0x7ff64815b7a0 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.900+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6480c8710 con 0x7ff64815b7a0 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.901+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6480689e0 con 0x7ff64815b7a0 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.901+0000 7ff63d7fa640 1 -- 172.21.15.12:0/3774641734 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff640048070 con 0x7ff64815b7a0 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.902+0000 7ff63d7fa640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff63406c8f0 0x7ff63406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.902+0000 7ff63d7fa640 1 -- 172.21.15.12:0/3774641734 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7ff6400ad510 con 0x7ff64815b7a0 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.902+0000 7ff64c91e640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff63406c8f0 0x7ff63406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:30.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.903+0000 7ff64c91e640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff63406c8f0 0x7ff63406edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff644009b00 tx=0x7ff644008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:30.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:30.906+0000 7ff63d7fa640 1 -- 172.21.15.12:0/3774641734 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff64007b4b0 con 0x7ff64815b7a0 2024-04-15T15:22:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.093+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff6480d11b0 con 0x7ff64815b7a0 2024-04-15T15:22:31.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.264+0000 7ff63d7fa640 1 -- 172.21.15.12:0/3774641734 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7ff64007f7b0 con 0x7ff64815b7a0 2024-04-15T15:22:31.266 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-04-15T15:22:31.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.266+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff63406c8f0 msgr2=0x7ff63406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:31.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.266+0000 7ff64eb82640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff63406c8f0 0x7ff63406edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff644009b00 tx=0x7ff644008040 comp rx=0 tx=0).stop 2024-04-15T15:22:31.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.266+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64815b7a0 msgr2=0x7ff64815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:31.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.266+0000 7ff64eb82640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64815b7a0 0x7ff64815bc20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff64000a9c0 tx=0x7ff640001dc0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.267+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 shutdown_connections 2024-04-15T15:22:31.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.267+0000 7ff64eb82640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff64815b7a0 0x7ff64815bc20 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.267+0000 7ff64eb82640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff6481f7100 0x7ff6480c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.267+0000 7ff64eb82640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff63406c8f0 0x7ff63406edb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.267+0000 7ff64eb82640 1 --2- 172.21.15.12:0/3774641734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630004d40 0x7ff6480ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.267+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 >> 172.21.15.12:0/3774641734 conn(0x7ff64808bf60 msgr2=0x7ff6480cb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.267+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 shutdown_connections 2024-04-15T15:22:31.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.267+0000 7ff64eb82640 1 -- 172.21.15.12:0/3774641734 wait complete. 2024-04-15T15:22:31.283 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-15T15:22:31.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.392+0000 7f997364a640 1 Processor -- start 2024-04-15T15:22:31.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.392+0000 7f997364a640 1 -- start start 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f997364a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c15e790 0x7f996c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f997364a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f996c0d1cf0 0x7f996c0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f997364a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f996c15eb70 con 0x7f996c0b7be0 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f997364a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f996c0d3790 con 0x7f996c0d1cf0 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f997364a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f996c0d45b0 con 0x7f996c15e790 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f9971be7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f996c0d1cf0 0x7f996c0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f9970be5640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f996c0b7be0 0x7f996c0d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60756/0 (socket says 172.21.15.12:60756) 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f9970be5640 1 -- 172.21.15.12:0/2995569337 learned_addr learned my addr 172.21.15.12:0/2995569337 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f99713e6640 1 --2- 172.21.15.12:0/2995569337 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c15e790 0x7f996c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f9971be7640 1 -- 172.21.15.12:0/2995569337 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c15e790 msgr2=0x7f996c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f9971be7640 1 --2- 172.21.15.12:0/2995569337 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c15e790 0x7f996c0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f9971be7640 1 -- 172.21.15.12:0/2995569337 >> v1:172.21.15.12:6789/0 conn(0x7f996c0b7be0 legacy=0x7f996c0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:31.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.393+0000 7f9971be7640 1 -- 172.21.15.12:0/2995569337 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996c0d3c80 con 0x7f996c0d1cf0 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.394+0000 7f9971be7640 1 --2- 172.21.15.12:0/2995569337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c0d1cf0 0x7f996c0d4a40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f996c0d3910 tx=0x7f996402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b339b1ca54ab8e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.394+0000 7f996bfff640 1 -- 172.21.15.12:0/2995569337 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964039070 con 0x7f996c0d1cf0 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.394+0000 7f996bfff640 1 -- 172.21.15.12:0/2995569337 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99640316f0 con 0x7f996c0d1cf0 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.394+0000 7f996bfff640 1 -- 172.21.15.12:0/2995569337 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964031ac0 con 0x7f996c0d1cf0 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.395+0000 7f997364a640 1 -- 172.21.15.12:0/2995569337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c0d1cf0 msgr2=0x7f996c0d4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.395+0000 7f997364a640 1 --2- 172.21.15.12:0/2995569337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c0d1cf0 0x7f996c0d4a40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f996c0d3910 tx=0x7f996402b790 comp rx=0 tx=0).stop 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.395+0000 7f997364a640 1 -- 172.21.15.12:0/2995569337 shutdown_connections 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.395+0000 7f997364a640 1 --2- 172.21.15.12:0/2995569337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c0d1cf0 0x7f996c0d4a40 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.395+0000 7f997364a640 1 --2- 172.21.15.12:0/2995569337 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c15e790 0x7f996c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.395+0000 7f997364a640 1 -- 172.21.15.12:0/2995569337 >> 172.21.15.12:0/2995569337 conn(0x7f996c067b40 msgr2=0x7f996c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:31.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.395+0000 7f997364a640 1 -- 172.21.15.12:0/2995569337 shutdown_connections 2024-04-15T15:22:31.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.395+0000 7f997364a640 1 -- 172.21.15.12:0/2995569337 wait complete. 2024-04-15T15:22:31.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.396+0000 7f997364a640 1 Processor -- start 2024-04-15T15:22:31.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.396+0000 7f997364a640 1 -- start start 2024-04-15T15:22:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f997364a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f996c0b7be0 0x7f996c14ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f997364a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c0d1cf0 0x7f996c0c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f997364a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c15e790 0x7f996c1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f997364a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f996c0d4f80 con 0x7f996c15e790 2024-04-15T15:22:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f997364a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f996c0d4240 con 0x7f996c0b7be0 2024-04-15T15:22:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f997364a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f996c0d5100 con 0x7f996c0d1cf0 2024-04-15T15:22:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9971be7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c15e790 0x7f996c1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:31.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9971be7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c15e790 0x7f996c1596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42600/0 (socket says 172.21.15.12:42600) 2024-04-15T15:22:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9971be7640 1 -- 172.21.15.12:0/3141709660 learned_addr learned my addr 172.21.15.12:0/3141709660 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f99713e6640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f996c0b7be0 0x7f996c14ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9970be5640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c0d1cf0 0x7f996c0c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9971be7640 1 -- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c0d1cf0 msgr2=0x7f996c0c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9971be7640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c0d1cf0 0x7f996c0c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9971be7640 1 -- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f996c0b7be0 msgr2=0x7f996c14ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9971be7640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f996c0b7be0 0x7f996c14ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:31.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9971be7640 1 -- 172.21.15.12:0/3141709660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996c14bf60 con 0x7f996c15e790 2024-04-15T15:22:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.397+0000 7f9970be5640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c0d1cf0 0x7f996c0c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.398+0000 7f9971be7640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c15e790 0x7f996c1596d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f996402b7d0 tx=0x7f99640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.398+0000 7f9969ffb640 1 -- 172.21.15.12:0/3141709660 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964043070 con 0x7f996c15e790 2024-04-15T15:22:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.398+0000 7f9969ffb640 1 -- 172.21.15.12:0/3141709660 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99640346a0 con 0x7f996c15e790 2024-04-15T15:22:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.398+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f996c0c8a90 con 0x7f996c15e790 2024-04-15T15:22:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.398+0000 7f9969ffb640 1 -- 172.21.15.12:0/3141709660 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964039040 con 0x7f996c15e790 2024-04-15T15:22:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.398+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f996c0c9a50 con 0x7f996c15e790 2024-04-15T15:22:31.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.399+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f996c0689e0 con 0x7f996c15e790 2024-04-15T15:22:31.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.400+0000 7f9969ffb640 1 -- 172.21.15.12:0/3141709660 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f996403c070 con 0x7f996c15e790 2024-04-15T15:22:31.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.400+0000 7f9969ffb640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f995806c8f0 0x7f995806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:31.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.400+0000 7f9969ffb640 1 -- 172.21.15.12:0/3141709660 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7f99640ad2b0 con 0x7f996c15e790 2024-04-15T15:22:31.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.400+0000 7f99713e6640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f995806c8f0 0x7f995806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:31.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.401+0000 7f99713e6640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f995806c8f0 0x7f995806edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f995c003aa0 tx=0x7f995c009830 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:31.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.405+0000 7f9969ffb640 1 -- 172.21.15.12:0/3141709660 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f996c14bf60 con 0x7f996c15e790 2024-04-15T15:22:31.591 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:22:31.594 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:22:31.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:31.592+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f996c0b7800 con 0x7f996c15e790 2024-04-15T15:22:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.274+0000 7f9969ffb640 1 -- 172.21.15.12:0/3141709660 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7f996407b290 con 0x7f996c15e790 2024-04-15T15:22:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-04-15T15:22:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.278+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f995806c8f0 msgr2=0x7f995806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.278+0000 7f997364a640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f995806c8f0 0x7f995806edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f995c003aa0 tx=0x7f995c009830 comp rx=0 tx=0).stop 2024-04-15T15:22:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.278+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c15e790 msgr2=0x7f996c1596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.278+0000 7f997364a640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c15e790 0x7f996c1596d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f996402b7d0 tx=0x7f99640046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.279+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 shutdown_connections 2024-04-15T15:22:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.279+0000 7f997364a640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f996c15e790 0x7f996c1596d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.279+0000 7f997364a640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f996c0d1cf0 0x7f996c0c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.279+0000 7f997364a640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f995806c8f0 0x7f995806edb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.279+0000 7f997364a640 1 --2- 172.21.15.12:0/3141709660 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f996c0b7be0 0x7f996c14ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.279+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 >> 172.21.15.12:0/3141709660 conn(0x7f996c067b40 msgr2=0x7f996c0d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:32.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.281+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 shutdown_connections 2024-04-15T15:22:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.281+0000 7f997364a640 1 -- 172.21.15.12:0/3141709660 wait complete. 2024-04-15T15:22:32.309 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-15T15:22:32.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.419+0000 7f3ee32d3640 1 Processor -- start 2024-04-15T15:22:32.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.419+0000 7f3ee32d3640 1 -- start start 2024-04-15T15:22:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee32d3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 0x7f3edc070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee32d3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3edc071d90 0x7f3edc0cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee32d3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3edc06ff30 con 0x7f3edc0ce8e0 2024-04-15T15:22:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee32d3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3edc070820 con 0x7f3edc071d90 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee32d3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3edc0cecc0 con 0x7f3edc09b0e0 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee106f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3edc0ce8e0 0x7f3edc09a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35912/0 (socket says 172.21.15.12:35912) 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee106f640 1 -- 172.21.15.12:0/3954786021 learned_addr learned my addr 172.21.15.12:0/3954786021 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee1870640 1 --2- 172.21.15.12:0/3954786021 >> v2:172.21.15.55:3300/0 conn(0x7f3edc071d90 0x7f3edc0cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee086e640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 0x7f3edc070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96930747 0 0) 0x7f3edc06ff30 con 0x7f3edc0ce8e0 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ec8003520 con 0x7f3edc0ce8e0 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2287790287 0 0) 0x7f3ec8003520 con 0x7f3edc0ce8e0 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 msgr2=0x7f3edc070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 0x7f3edc070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 >> v2:172.21.15.55:3300/0 conn(0x7f3edc071d90 msgr2=0x7f3edc0cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 --2- 172.21.15.12:0/3954786021 >> v2:172.21.15.55:3300/0 conn(0x7f3edc071d90 0x7f3edc0cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc09b9d0 con 0x7f3edc0ce8e0 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f3ed8001690 con 0x7f3edc0ce8e0 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ee1870640 1 --2- 172.21.15.12:0/3954786021 >> v2:172.21.15.55:3300/0 conn(0x7f3edc071d90 0x7f3edc0cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 >> v1:172.21.15.12:6789/0 conn(0x7f3edc0ce8e0 legacy=0x7f3edc09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.420+0000 7f3ed3fff640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3ec8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ed3fff640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ec8004800 0x7f3ec8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ed3fff640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 0x7f3ec80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc070820 con 0x7f3ec80039b0 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc0cecc0 con 0x7f3ec8004800 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec800b9f0 con 0x7f3edc09b0e0 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee106f640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3ec8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee086e640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ec8004800 0x7f3ec8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee1870640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 0x7f3ec80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee106f640 1 -- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 msgr2=0x7f3ec80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee106f640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 0x7f3ec80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee106f640 1 -- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ec8004800 msgr2=0x7f3ec8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee106f640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ec8004800 0x7f3ec8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee106f640 1 -- 172.21.15.12:0/3954786021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec8003520 con 0x7f3ec80039b0 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee1870640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc09b0e0 0x7f3ec80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.421+0000 7f3ee086e640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ec8004800 0x7f3ec8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ee106f640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3ec8003dd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c210 tx=0x7f3ec8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3bcbadd6855826 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8007d70 con 0x7f3ec80039b0 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed8030070 con 0x7f3ec80039b0 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ed3fff640 1 -- 172.21.15.12:0/3954786021 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed802cd60 con 0x7f3ec80039b0 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ee32d3640 1 -- 172.21.15.12:0/3954786021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 msgr2=0x7f3ec8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ee32d3640 1 --2- 172.21.15.12:0/3954786021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3ec8003dd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c210 tx=0x7f3ec8005830 comp rx=0 tx=0).stop 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ee1870640 1 -- 172.21.15.12:0/3954786021 reap_dead start 2024-04-15T15:22:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ee32d3640 1 -- 172.21.15.12:0/3954786021 shutdown_connections 2024-04-15T15:22:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.422+0000 7f3ee32d3640 1 -- 172.21.15.12:0/3954786021 >> 172.21.15.12:0/3954786021 conn(0x7f3edc0b3f40 msgr2=0x7f3edc0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.423+0000 7f3ee32d3640 1 -- 172.21.15.12:0/3954786021 shutdown_connections 2024-04-15T15:22:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.423+0000 7f3ee32d3640 1 -- 172.21.15.12:0/3954786021 wait complete. 2024-04-15T15:22:32.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.424+0000 7f3ee32d3640 1 Processor -- start 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.424+0000 7f3ee32d3640 1 -- start start 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.424+0000 7f3ee32d3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3edc1dc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.424+0000 7f3ee32d3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc1e7b70 0x7f3edc1e7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.424+0000 7f3ee32d3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3edc1f7cf0 0x7f3edc1f8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.424+0000 7f3ee32d3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc0704b0 con 0x7f3ec80039b0 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.424+0000 7f3ee32d3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc0701d0 con 0x7f3edc1f7cf0 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.424+0000 7f3ee32d3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3edc071970 con 0x7f3edc1e7b70 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3edc1dc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3edc1dc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49388/0 (socket says 172.21.15.12:49388) 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 -- 172.21.15.12:0/737236467 learned_addr learned my addr 172.21.15.12:0/737236467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee1870640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3edc1f7cf0 0x7f3edc1f8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee086e640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc1e7b70 0x7f3edc1e7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 -- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc1e7b70 msgr2=0x7f3edc1e7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:32.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc1e7b70 0x7f3edc1e7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 -- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3edc1f7cf0 msgr2=0x7f3edc1f8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3edc1f7cf0 0x7f3edc1f8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 -- 172.21.15.12:0/737236467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc1f8860 con 0x7f3ec80039b0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee1870640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3edc1f7cf0 0x7f3edc1f8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee086e640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc1e7b70 0x7f3edc1e7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee106f640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3edc1dc520 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3ed802c5d0 tx=0x7f3ed8001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ed1ffb640 1 -- 172.21.15.12:0/737236467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8045070 con 0x7f3ec80039b0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ed1ffb640 1 -- 172.21.15.12:0/737236467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed8004bf0 con 0x7f3ec80039b0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3edc1f8b40 con 0x7f3ec80039b0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.425+0000 7f3ed1ffb640 1 -- 172.21.15.12:0/737236467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8001cb0 con 0x7f3ec80039b0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.426+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3edc1e7370 con 0x7f3ec80039b0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.426+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3edc0689e0 con 0x7f3ec80039b0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.427+0000 7f3ed1ffb640 1 -- 172.21.15.12:0/737236467 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3ed8007070 con 0x7f3ec80039b0 2024-04-15T15:22:32.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.427+0000 7f3ed1ffb640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ec006c8f0 0x7f3ec006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:32.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.428+0000 7f3ed1ffb640 1 -- 172.21.15.12:0/737236467 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7f3ed80acd70 con 0x7f3ec80039b0 2024-04-15T15:22:32.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.428+0000 7f3ee086e640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ec006c8f0 0x7f3ec006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:32.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.428+0000 7f3ee086e640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ec006c8f0 0x7f3ec006edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc006f80 tx=0x7f3ecc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:32.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.432+0000 7f3ed1ffb640 1 -- 172.21.15.12:0/737236467 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3ed807ab90 con 0x7f3ec80039b0 2024-04-15T15:22:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:32.654+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3edc0b4ca0 con 0x7f3ec80039b0 2024-04-15T15:22:33.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.287+0000 7f3ed1ffb640 1 -- 172.21.15.12:0/737236467 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7f3ed807ee90 con 0x7f3ec80039b0 2024-04-15T15:22:33.289 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 5 and data pool 6 2024-04-15T15:22:33.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.292+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ec006c8f0 msgr2=0x7f3ec006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ec006c8f0 0x7f3ec006edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc006f80 tx=0x7f3ecc008040 comp rx=0 tx=0).stop 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 msgr2=0x7f3edc1dc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3edc1dc520 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3ed802c5d0 tx=0x7f3ed8001a60 comp rx=0 tx=0).stop 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 shutdown_connections 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ec006c8f0 0x7f3ec006edb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3edc1f7cf0 0x7f3edc1f8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3edc1e7b70 0x7f3edc1e7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 --2- 172.21.15.12:0/737236467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ec80039b0 0x7f3edc1dc520 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.293+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 >> 172.21.15.12:0/737236467 conn(0x7f3edc0b3f40 msgr2=0x7f3edc070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.294+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 shutdown_connections 2024-04-15T15:22:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.294+0000 7f3ee32d3640 1 -- 172.21.15.12:0/737236467 wait complete. 2024-04-15T15:22:33.318 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.424+0000 7f2f4dfd4640 1 Processor -- start 2024-04-15T15:22:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.424+0000 7f2f4dfd4640 1 -- start start 2024-04-15T15:22:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f4dfd4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f48072d80 0x7f2f48071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f4dfd4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480d0da0 0x7f2f480af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f4dfd4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f4815db40 con 0x7f2f480af3e0 2024-04-15T15:22:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f4dfd4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2f4815cfc0 con 0x7f2f48072d80 2024-04-15T15:22:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f4dfd4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4815d2a0 con 0x7f2f480d0da0 2024-04-15T15:22:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f477fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f48072d80 0x7f2f48071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480d0da0 0x7f2f480af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f46ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2f480af3e0 0x7f2f480d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:45468/0 (socket says 172.21.15.12:45468) 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f46ffd640 1 -- 172.21.15.12:0/383770404 learned_addr learned my addr 172.21.15.12:0/383770404 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480d0da0 0x7f2f480af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:42672/0 (socket says 172.21.15.12:42672) 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.425+0000 7f2f477fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f48072d80 0x7f2f48071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:40512/0 (socket says 172.21.15.12:40512) 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.426+0000 7f2f467fc640 1 -- 172.21.15.12:0/383770404 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497972442 0 0) 0x7f2f4815db40 con 0x7f2f480af3e0 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.426+0000 7f2f467fc640 1 -- 172.21.15.12:0/383770404 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f30003520 con 0x7f2f480af3e0 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.426+0000 7f2f47fff640 1 -- 172.21.15.12:0/383770404 >> v2:172.21.15.55:3300/0 conn(0x7f2f48072d80 msgr2=0x7f2f48071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.426+0000 7f2f47fff640 1 --2- 172.21.15.12:0/383770404 >> v2:172.21.15.55:3300/0 conn(0x7f2f48072d80 0x7f2f48071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.426+0000 7f2f47fff640 1 -- 172.21.15.12:0/383770404 >> v1:172.21.15.55:6789/0 conn(0x7f2f480af3e0 legacy=0x7f2f480d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.426+0000 7f2f47fff640 1 -- 172.21.15.12:0/383770404 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4815d580 con 0x7f2f480d0da0 2024-04-15T15:22:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.426+0000 7f2f477fe640 1 --2- 172.21.15.12:0/383770404 >> v2:172.21.15.55:3300/0 conn(0x7f2f48072d80 0x7f2f48071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.426+0000 7f2f47fff640 1 --2- 172.21.15.12:0/383770404 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480d0da0 0x7f2f480af840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009d90 tx=0x7f2f4002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc34d9b9f2b4f29a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f467fc640 1 -- 172.21.15.12:0/383770404 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40039070 con 0x7f2f480d0da0 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f467fc640 1 -- 172.21.15.12:0/383770404 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f4002cbc0 con 0x7f2f480d0da0 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f467fc640 1 -- 172.21.15.12:0/383770404 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40034440 con 0x7f2f480d0da0 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/383770404 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480d0da0 msgr2=0x7f2f480af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/383770404 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480d0da0 0x7f2f480af840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009d90 tx=0x7f2f4002be10 comp rx=0 tx=0).stop 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/383770404 shutdown_connections 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/383770404 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480d0da0 0x7f2f480af840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/383770404 >> v2:172.21.15.55:3300/0 conn(0x7f2f48072d80 0x7f2f48071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/383770404 >> 172.21.15.12:0/383770404 conn(0x7f2f4806be00 msgr2=0x7f2f4806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/383770404 shutdown_connections 2024-04-15T15:22:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.427+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/383770404 wait complete. 2024-04-15T15:22:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.428+0000 7f2f4dfd4640 1 Processor -- start 2024-04-15T15:22:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f4dfd4640 1 -- start start 2024-04-15T15:22:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f4dfd4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f48072d80 0x7f2f4814e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f4dfd4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480af3e0 0x7f2f481598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f4dfd4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480d0da0 0x7f2f4815b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f4dfd4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4815cd10 con 0x7f2f48072d80 2024-04-15T15:22:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f4dfd4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4815cb90 con 0x7f2f480d0da0 2024-04-15T15:22:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f4dfd4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4806e060 con 0x7f2f480af3e0 2024-04-15T15:22:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f477fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f48072d80 0x7f2f4814e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f477fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f48072d80 0x7f2f4814e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49402/0 (socket says 172.21.15.12:49402) 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f477fe640 1 -- 172.21.15.12:0/396991473 learned_addr learned my addr 172.21.15.12:0/396991473 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.429+0000 7f2f46ffd640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480af3e0 0x7f2f481598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f47fff640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480d0da0 0x7f2f4815b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f477fe640 1 -- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480af3e0 msgr2=0x7f2f481598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f477fe640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480af3e0 0x7f2f481598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f477fe640 1 -- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480d0da0 msgr2=0x7f2f4815b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f477fe640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480d0da0 0x7f2f4815b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f477fe640 1 -- 172.21.15.12:0/396991473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f480ca5d0 con 0x7f2f48072d80 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f477fe640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f48072d80 0x7f2f4814e5c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800ec10 tx=0x7f2f3800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f2ffff640 1 -- 172.21.15.12:0/396991473 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38017070 con 0x7f2f48072d80 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f2ffff640 1 -- 172.21.15.12:0/396991473 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f3800cc30 con 0x7f2f48072d80 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f480ca8b0 con 0x7f2f48072d80 2024-04-15T15:22:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f2ffff640 1 -- 172.21.15.12:0/396991473 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38013470 con 0x7f2f48072d80 2024-04-15T15:22:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.430+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f480cae70 con 0x7f2f48072d80 2024-04-15T15:22:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.431+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f480689e0 con 0x7f2f48072d80 2024-04-15T15:22:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.431+0000 7f2f2ffff640 1 -- 172.21.15.12:0/396991473 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f3801a070 con 0x7f2f48072d80 2024-04-15T15:22:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.432+0000 7f2f2ffff640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f3406c920 0x7f2f3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.432+0000 7f2f2ffff640 1 -- 172.21.15.12:0/396991473 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f2f3808c090 con 0x7f2f48072d80 2024-04-15T15:22:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.432+0000 7f2f46ffd640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f3406c920 0x7f2f3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.433+0000 7f2f46ffd640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f3406c920 0x7f2f3406ede0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2f481f79b0 tx=0x7f2f3c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:33.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.437+0000 7f2f2ffff640 1 -- 172.21.15.12:0/396991473 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f3805a220 con 0x7f2f48072d80 2024-04-15T15:22:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.621+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f4815a4b0 con 0x7f2f48072d80 2024-04-15T15:22:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.622+0000 7f2f2ffff640 1 -- 172.21.15.12:0/396991473 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16919 (secure 0 0 0) 0x7f2f38059b20 con 0x7f2f48072d80 2024-04-15T15:22:33.625 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:33.625 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":38,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:22:32.656679+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f3406c920 msgr2=0x7f2f3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f3406c920 0x7f2f3406ede0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2f481f79b0 tx=0x7f2f3c009040 comp rx=0 tx=0).stop 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f48072d80 msgr2=0x7f2f4814e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f48072d80 0x7f2f4814e5c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800ec10 tx=0x7f2f3800c520 comp rx=0 tx=0).stop 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 shutdown_connections 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f3406c920 0x7f2f3406ede0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480d0da0 0x7f2f4815b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f480af3e0 0x7f2f481598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 --2- 172.21.15.12:0/396991473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f48072d80 0x7f2f4814e5c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 >> 172.21.15.12:0/396991473 conn(0x7f2f4806be00 msgr2=0x7f2f48071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 shutdown_connections 2024-04-15T15:22:33.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.624+0000 7f2f4dfd4640 1 -- 172.21.15.12:0/396991473 wait complete. 2024-04-15T15:22:33.638 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:33.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.739+0000 7fe850d52640 1 Processor -- start 2024-04-15T15:22:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.739+0000 7fe850d52640 1 -- start start 2024-04-15T15:22:33.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.740+0000 7fe850d52640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe84c15e700 0x7fe84c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.740+0000 7fe850d52640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe84c0d2360 0x7fe84c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.740+0000 7fe850d52640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe84c0b47a0 con 0x7fe84c15eae0 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.740+0000 7fe850d52640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe84c0b53b0 con 0x7fe84c15e700 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.740+0000 7fe850d52640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe84c0b4a80 con 0x7fe84c0d2360 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.740+0000 7fe84a59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe84c15e700 0x7fe84c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.740+0000 7fe84a59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe84c15e700 0x7fe84c0b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49412/0 (socket says 172.21.15.12:49412) 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.740+0000 7fe84a59c640 1 -- 172.21.15.12:0/2504199240 learned_addr learned my addr 172.21.15.12:0/2504199240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.741+0000 7fe849d9b640 1 --2- 172.21.15.12:0/2504199240 >> v2:172.21.15.55:3300/0 conn(0x7fe84c0d2360 0x7fe84c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.741+0000 7fe84a59c640 1 -- 172.21.15.12:0/2504199240 >> v2:172.21.15.55:3300/0 conn(0x7fe84c0d2360 msgr2=0x7fe84c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.741+0000 7fe84a59c640 1 --2- 172.21.15.12:0/2504199240 >> v2:172.21.15.55:3300/0 conn(0x7fe84c0d2360 0x7fe84c0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.741+0000 7fe84a59c640 1 -- 172.21.15.12:0/2504199240 >> v1:172.21.15.55:6789/0 conn(0x7fe84c15eae0 legacy=0x7fe84c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.741+0000 7fe84a59c640 1 -- 172.21.15.12:0/2504199240 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c0b4d60 con 0x7fe84c15e700 2024-04-15T15:22:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.741+0000 7fe849d9b640 1 --2- 172.21.15.12:0/2504199240 >> v2:172.21.15.55:3300/0 conn(0x7fe84c0d2360 0x7fe84c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe84a59c640 1 --2- 172.21.15.12:0/2504199240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b5840 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe84c08c7b0 tx=0x7fe83c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f822b3517e5b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe84959a640 1 -- 172.21.15.12:0/2504199240 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c02cb70 con 0x7fe84c15e700 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe84959a640 1 -- 172.21.15.12:0/2504199240 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe83c02cd10 con 0x7fe84c15e700 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe84959a640 1 -- 172.21.15.12:0/2504199240 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c034610 con 0x7fe84c15e700 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe850d52640 1 -- 172.21.15.12:0/2504199240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 msgr2=0x7fe84c0b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe850d52640 1 --2- 172.21.15.12:0/2504199240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b5840 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe84c08c7b0 tx=0x7fe83c02bfd0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe850d52640 1 -- 172.21.15.12:0/2504199240 shutdown_connections 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe850d52640 1 --2- 172.21.15.12:0/2504199240 >> v2:172.21.15.55:3300/0 conn(0x7fe84c0d2360 0x7fe84c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe850d52640 1 --2- 172.21.15.12:0/2504199240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b5840 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe850d52640 1 -- 172.21.15.12:0/2504199240 >> 172.21.15.12:0/2504199240 conn(0x7fe84c08bf60 msgr2=0x7fe84c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe850d52640 1 -- 172.21.15.12:0/2504199240 shutdown_connections 2024-04-15T15:22:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.742+0000 7fe850d52640 1 -- 172.21.15.12:0/2504199240 wait complete. 2024-04-15T15:22:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.744+0000 7fe850d52640 1 Processor -- start 2024-04-15T15:22:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.744+0000 7fe850d52640 1 -- start start 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.744+0000 7fe850d52640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe84c0d2360 0x7fe84c1f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.744+0000 7fe850d52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.744+0000 7fe850d52640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe84c15eae0 0x7fe84c0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.744+0000 7fe850d52640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c0b5d80 con 0x7fe84c15e700 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.744+0000 7fe850d52640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c0b4450 con 0x7fe84c15eae0 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.744+0000 7fe850d52640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe84c0d1d40 con 0x7fe84c0d2360 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49420/0 (socket says 172.21.15.12:49420) 2024-04-15T15:22:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 -- 172.21.15.12:0/149350401 learned_addr learned my addr 172.21.15.12:0/149350401 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe84a59c640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe84c0d2360 0x7fe84c1f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe84ad9d640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe84c15eae0 0x7fe84c0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 -- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe84c0d2360 msgr2=0x7fe84c1f48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe84c0d2360 0x7fe84c1f48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 -- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe84c15eae0 msgr2=0x7fe84c0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe84c15eae0 0x7fe84c0c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 -- 172.21.15.12:0/149350401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c0b3b50 con 0x7fe84c15e700 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe849d9b640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b3610 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe83400e7e0 tx=0x7fe83400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe842ffd640 1 -- 172.21.15.12:0/149350401 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83400f040 con 0x7fe84c15e700 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe842ffd640 1 -- 172.21.15.12:0/149350401 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe834004030 con 0x7fe84c15e700 2024-04-15T15:22:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe842ffd640 1 -- 172.21.15.12:0/149350401 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834004400 con 0x7fe84c15e700 2024-04-15T15:22:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84c0b2270 con 0x7fe84c15e700 2024-04-15T15:22:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.745+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84c0b2830 con 0x7fe84c15e700 2024-04-15T15:22:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.746+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe84c0d10c0 con 0x7fe84c15e700 2024-04-15T15:22:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.747+0000 7fe842ffd640 1 -- 172.21.15.12:0/149350401 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe83400c6d0 con 0x7fe84c15e700 2024-04-15T15:22:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.747+0000 7fe842ffd640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe82c06c8f0 0x7fe82c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.747+0000 7fe842ffd640 1 -- 172.21.15.12:0/149350401 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fe83408bab0 con 0x7fe84c15e700 2024-04-15T15:22:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.748+0000 7fe84a59c640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe82c06c8f0 0x7fe82c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.748+0000 7fe84a59c640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe82c06c8f0 0x7fe82c06edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe83c035000 tx=0x7fe83c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.752+0000 7fe842ffd640 1 -- 172.21.15.12:0/149350401 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe84c0d10c0 con 0x7fe84c15e700 2024-04-15T15:22:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.936+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe84c0689e0 con 0x7fe84c15e700 2024-04-15T15:22:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.938+0000 7fe842ffd640 1 -- 172.21.15.12:0/149350401 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16919 (secure 0 0 0) 0x7fe834017020 con 0x7fe84c15e700 2024-04-15T15:22:33.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:33.940 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":38,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:22:32.656679+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe82c06c8f0 msgr2=0x7fe82c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe82c06c8f0 0x7fe82c06edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe83c035000 tx=0x7fe83c030000 comp rx=0 tx=0).stop 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 msgr2=0x7fe84c0b3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b3610 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe83400e7e0 tx=0x7fe83400eca0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 shutdown_connections 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe82c06c8f0 0x7fe82c06edb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe84c15eae0 0x7fe84c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe84c15e700 0x7fe84c0b3610 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 --2- 172.21.15.12:0/149350401 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe84c0d2360 0x7fe84c1f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 >> 172.21.15.12:0/149350401 conn(0x7fe84c08bf60 msgr2=0x7fe84c0d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 shutdown_connections 2024-04-15T15:22:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:33.940+0000 7fe850d52640 1 -- 172.21.15.12:0/149350401 wait complete. 2024-04-15T15:22:33.953 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-15T15:22:34.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.058+0000 7f0d36f8f640 1 Processor -- start 2024-04-15T15:22:34.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.059+0000 7f0d36f8f640 1 -- start start 2024-04-15T15:22:34.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.059+0000 7f0d36f8f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0d300d2360 0x7f0d300d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:34.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.059+0000 7f0d36f8f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0d3015eae0 0x7f0d300d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:34.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.059+0000 7f0d36f8f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d300b47a0 con 0x7f0d3015e700 2024-04-15T15:22:34.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.059+0000 7f0d36f8f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0d300b53b0 con 0x7f0d300d2360 2024-04-15T15:22:34.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.059+0000 7f0d36f8f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0d300b4a80 con 0x7f0d3015eae0 2024-04-15T15:22:34.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.059+0000 7f0d2ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0d300d2360 0x7f0d300d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:34.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.060+0000 7f0d34d2b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0d3015e700 0x7f0d300b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35928/0 (socket says 172.21.15.12:35928) 2024-04-15T15:22:34.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.060+0000 7f0d34d2b640 1 -- 172.21.15.12:0/2241380022 learned_addr learned my addr 172.21.15.12:0/2241380022 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:34.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.060+0000 7f0d2ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0d300d2360 0x7f0d300d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49432/0 (socket says 172.21.15.12:49432) 2024-04-15T15:22:34.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.060+0000 7f0d2ffff640 1 -- 172.21.15.12:0/2241380022 >> v2:172.21.15.55:3300/0 conn(0x7f0d3015eae0 msgr2=0x7f0d300d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:34.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.060+0000 7f0d2ffff640 1 --2- 172.21.15.12:0/2241380022 >> v2:172.21.15.55:3300/0 conn(0x7f0d3015eae0 0x7f0d300d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:34.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.060+0000 7f0d2ffff640 1 -- 172.21.15.12:0/2241380022 >> v1:172.21.15.12:6789/0 conn(0x7f0d3015e700 legacy=0x7f0d300b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:34.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.060+0000 7f0d2ffff640 1 -- 172.21.15.12:0/2241380022 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d300b4d60 con 0x7f0d300d2360 2024-04-15T15:22:34.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d2ffff640 1 --2- 172.21.15.12:0/2241380022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300d2b60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0d300d1b00 tx=0x7f0d2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f565106da7b1e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:34.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d2f7fe640 1 -- 172.21.15.12:0/2241380022 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d20035020 con 0x7f0d300d2360 2024-04-15T15:22:34.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d2f7fe640 1 -- 172.21.15.12:0/2241380022 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d20031830 con 0x7f0d300d2360 2024-04-15T15:22:34.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d2f7fe640 1 -- 172.21.15.12:0/2241380022 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d20031c00 con 0x7f0d300d2360 2024-04-15T15:22:34.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2241380022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 msgr2=0x7f0d300d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:34.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2241380022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300d2b60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0d300d1b00 tx=0x7f0d2002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:34.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2241380022 shutdown_connections 2024-04-15T15:22:34.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2241380022 >> v2:172.21.15.55:3300/0 conn(0x7f0d3015eae0 0x7f0d300d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:34.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2241380022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300d2b60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:34.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2241380022 >> 172.21.15.12:0/2241380022 conn(0x7f0d3008bf60 msgr2=0x7f0d300d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:34.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2241380022 shutdown_connections 2024-04-15T15:22:34.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.061+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2241380022 wait complete. 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.062+0000 7f0d36f8f640 1 Processor -- start 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.062+0000 7f0d36f8f640 1 -- start start 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d36f8f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d36f8f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0d3015e700 0x7f0d300b8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d36f8f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d3015eae0 0x7f0d300b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d36f8f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0d300d1d40 con 0x7f0d300d2360 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d36f8f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0d300b4450 con 0x7f0d3015eae0 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d36f8f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0d300d1ec0 con 0x7f0d3015e700 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d34d2b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:34.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d34d2b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49440/0 (socket says 172.21.15.12:49440) 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.063+0000 7f0d34d2b640 1 -- 172.21.15.12:0/2375860281 learned_addr learned my addr 172.21.15.12:0/2375860281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d34d2b640 1 -- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0d3015e700 msgr2=0x7f0d300b8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d34d2b640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0d3015e700 0x7f0d300b8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d34d2b640 1 -- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d3015eae0 msgr2=0x7f0d300b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d34d2b640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d3015eae0 0x7f0d300b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d34d2b640 1 -- 172.21.15.12:0/2375860281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d300cd630 con 0x7f0d300d2360 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d34d2b640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300ce120 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0d2800c780 tx=0x7f0d2800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d2d7fa640 1 -- 172.21.15.12:0/2375860281 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28017070 con 0x7f0d300d2360 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d2d7fa640 1 -- 172.21.15.12:0/2375860281 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d2800a640 con 0x7f0d300d2360 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d2d7fa640 1 -- 172.21.15.12:0/2375860281 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2800aa50 con 0x7f0d300d2360 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d301f80c0 con 0x7f0d300d2360 2024-04-15T15:22:34.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.064+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d301f8680 con 0x7f0d300d2360 2024-04-15T15:22:34.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.065+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d300689e0 con 0x7f0d300d2360 2024-04-15T15:22:34.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.065+0000 7f0d2d7fa640 1 -- 172.21.15.12:0/2375860281 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d2801a070 con 0x7f0d300d2360 2024-04-15T15:22:34.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.066+0000 7f0d2d7fa640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0d1006c8f0 0x7f0d1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:34.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.066+0000 7f0d2d7fa640 1 -- 172.21.15.12:0/2375860281 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f0d2808b2e0 con 0x7f0d300d2360 2024-04-15T15:22:34.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.066+0000 7f0d2ffff640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0d1006c8f0 0x7f0d1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:34.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.066+0000 7f0d2ffff640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0d1006c8f0 0x7f0d1006edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0d20036000 tx=0x7f0d20030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:34.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.070+0000 7f0d2d7fa640 1 -- 172.21.15.12:0/2375860281 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d28059610 con 0x7f0d300d2360 2024-04-15T15:22:34.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:34.294+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0d300d1340 con 0x7f0d300d2360 2024-04-15T15:22:35.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.299+0000 7f0d2d7fa640 1 -- 172.21.15.12:0/2375860281 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f0d28058f10 con 0x7f0d300d2360 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0d1006c8f0 msgr2=0x7f0d1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0d1006c8f0 0x7f0d1006edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0d20036000 tx=0x7f0d20030000 comp rx=0 tx=0).stop 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 msgr2=0x7f0d300ce120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300ce120 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0d2800c780 tx=0x7f0d2800cc40 comp rx=0 tx=0).stop 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 shutdown_connections 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0d1006c8f0 0x7f0d1006edb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0d3015eae0 0x7f0d300b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0d3015e700 0x7f0d300b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 --2- 172.21.15.12:0/2375860281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0d300d2360 0x7f0d300ce120 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 >> 172.21.15.12:0/2375860281 conn(0x7f0d3008bf60 msgr2=0x7f0d300d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 shutdown_connections 2024-04-15T15:22:35.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.302+0000 7f0d36f8f640 1 -- 172.21.15.12:0/2375860281 wait complete. 2024-04-15T15:22:35.318 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-15T15:22:35.318 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-15T15:22:35.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.430+0000 7fb348cb1640 1 Processor -- start 2024-04-15T15:22:35.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.430+0000 7fb348cb1640 1 -- start start 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb348cb1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb34415ecb0 0x7fb3440b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb348cb1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3440b10d0 con 0x7fb3440b1d30 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb348cb1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3440af920 con 0x7fb3440b4350 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb348cb1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb34415f090 con 0x7fb34415ecb0 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb34259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb34415ecb0 0x7fb3440b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb34259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb34415ecb0 0x7fb3440b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49450/0 (socket says 172.21.15.12:49450) 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb34259c640 1 -- 172.21.15.12:0/166796580 learned_addr learned my addr 172.21.15.12:0/166796580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb34259c640 1 -- 172.21.15.12:0/166796580 >> v1:172.21.15.55:6789/0 conn(0x7fb3440b4350 legacy=0x7fb3440b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb34259c640 1 -- 172.21.15.12:0/166796580 >> v1:172.21.15.12:6789/0 conn(0x7fb3440b1d30 legacy=0x7fb3440ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:35.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.431+0000 7fb34259c640 1 -- 172.21.15.12:0/166796580 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3440afc00 con 0x7fb34415ecb0 2024-04-15T15:22:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.432+0000 7fb34259c640 1 --2- 172.21.15.12:0/166796580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34415ecb0 0x7fb3440b0b90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb3340098a0 tx=0x7fb33402baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4367ab1288833696 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.432+0000 7fb341d9b640 1 -- 172.21.15.12:0/166796580 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33402d860 con 0x7fb34415ecb0 2024-04-15T15:22:35.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.432+0000 7fb341d9b640 1 -- 172.21.15.12:0/166796580 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb33402da00 con 0x7fb34415ecb0 2024-04-15T15:22:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.432+0000 7fb341d9b640 1 -- 172.21.15.12:0/166796580 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33402ddd0 con 0x7fb34415ecb0 2024-04-15T15:22:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.433+0000 7fb348cb1640 1 -- 172.21.15.12:0/166796580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34415ecb0 msgr2=0x7fb3440b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.433+0000 7fb348cb1640 1 --2- 172.21.15.12:0/166796580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34415ecb0 0x7fb3440b0b90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb3340098a0 tx=0x7fb33402baa0 comp rx=0 tx=0).stop 2024-04-15T15:22:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.433+0000 7fb348cb1640 1 -- 172.21.15.12:0/166796580 shutdown_connections 2024-04-15T15:22:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.433+0000 7fb348cb1640 1 --2- 172.21.15.12:0/166796580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb34415ecb0 0x7fb3440b0b90 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.433+0000 7fb348cb1640 1 -- 172.21.15.12:0/166796580 >> 172.21.15.12:0/166796580 conn(0x7fb3440b6870 msgr2=0x7fb3440b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.433+0000 7fb348cb1640 1 -- 172.21.15.12:0/166796580 shutdown_connections 2024-04-15T15:22:35.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.433+0000 7fb348cb1640 1 -- 172.21.15.12:0/166796580 wait complete. 2024-04-15T15:22:35.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.434+0000 7fb348cb1640 1 Processor -- start 2024-04-15T15:22:35.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.434+0000 7fb348cb1640 1 -- start start 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb348cb1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3440b1d30 0x7fb344149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb348cb1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3440b4350 0x7fb34415f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb348cb1640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb34415ecb0 0x7fb34415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb348cb1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3440b47c0 con 0x7fb3440b1d30 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb348cb1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb3440cdf80 con 0x7fb3440b4350 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb348cb1640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb3441608f0 con 0x7fb34415ecb0 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb34259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3440b1d30 0x7fb344149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb34259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3440b1d30 0x7fb344149e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49454/0 (socket says 172.21.15.12:49454) 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb34259c640 1 -- 172.21.15.12:0/3756078718 learned_addr learned my addr 172.21.15.12:0/3756078718 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:35.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb342d9d640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb34415ecb0 0x7fb34415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb33bfff640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3440b4350 0x7fb34415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb34259c640 1 -- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb34415ecb0 msgr2=0x7fb34415c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb34259c640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb34415ecb0 0x7fb34415c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb34259c640 1 -- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3440b4350 msgr2=0x7fb34415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb34259c640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3440b4350 0x7fb34415f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.435+0000 7fb34259c640 1 -- 172.21.15.12:0/3756078718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34415fad0 con 0x7fb3440b1d30 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.436+0000 7fb34259c640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3440b1d30 0x7fb344149e90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb3340099c0 tx=0x7fb334004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.436+0000 7fb33affd640 1 -- 172.21.15.12:0/3756078718 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334047070 con 0x7fb3440b1d30 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.436+0000 7fb33affd640 1 -- 172.21.15.12:0/3756078718 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb334037a40 con 0x7fb3440b1d30 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.436+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb34415fdb0 con 0x7fb3440b1d30 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.436+0000 7fb33affd640 1 -- 172.21.15.12:0/3756078718 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334037e10 con 0x7fb3440b1d30 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.436+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb344160b40 con 0x7fb3440b1d30 2024-04-15T15:22:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.437+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3440689e0 con 0x7fb3440b1d30 2024-04-15T15:22:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.437+0000 7fb33affd640 1 -- 172.21.15.12:0/3756078718 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb334041390 con 0x7fb3440b1d30 2024-04-15T15:22:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.438+0000 7fb33affd640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb32406c8f0 0x7fb32406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.438+0000 7fb33affd640 1 -- 172.21.15.12:0/3756078718 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fb3340ad850 con 0x7fb3440b1d30 2024-04-15T15:22:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.438+0000 7fb33bfff640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb32406c8f0 0x7fb32406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:35.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.438+0000 7fb33bfff640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb32406c8f0 0x7fb32406edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb34415dd00 tx=0x7fb32c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.442+0000 7fb33affd640 1 -- 172.21.15.12:0/3756078718 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb33407b530 con 0x7fb3440b1d30 2024-04-15T15:22:35.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:35.665+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fb34415b1d0 con 0x7fb3440b1d30 2024-04-15T15:22:36.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.301+0000 7fb33affd640 1 -- 172.21.15.12:0/3756078718 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fb33407f830 con 0x7fb3440b1d30 2024-04-15T15:22:36.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb32406c8f0 msgr2=0x7fb32406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb32406c8f0 0x7fb32406edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb34415dd00 tx=0x7fb32c009040 comp rx=0 tx=0).stop 2024-04-15T15:22:36.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3440b1d30 msgr2=0x7fb344149e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3440b1d30 0x7fb344149e90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb3340099c0 tx=0x7fb334004730 comp rx=0 tx=0).stop 2024-04-15T15:22:36.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 shutdown_connections 2024-04-15T15:22:36.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb34415ecb0 0x7fb34415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb32406c8f0 0x7fb32406edb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3440b4350 0x7fb34415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 --2- 172.21.15.12:0/3756078718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3440b1d30 0x7fb344149e90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 >> 172.21.15.12:0/3756078718 conn(0x7fb3440b6870 msgr2=0x7fb3440b5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:36.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 shutdown_connections 2024-04-15T15:22:36.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.304+0000 7fb348cb1640 1 -- 172.21.15.12:0/3756078718 wait complete. 2024-04-15T15:22:36.320 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-15T15:22:36.320 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:22:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.428+0000 7f3cc1bbd640 1 Processor -- start 2024-04-15T15:22:36.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.428+0000 7f3cc1bbd640 1 -- start start 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.429+0000 7f3cc1bbd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3cbc0d2360 0x7f3cbc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.429+0000 7f3cc1bbd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc15eae0 0x7f3cbc0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.429+0000 7f3cc1bbd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cbc0b47a0 con 0x7f3cbc15e700 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.429+0000 7f3cc1bbd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3cbc0b53b0 con 0x7f3cbc0d2360 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.429+0000 7f3cc1bbd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3cbc0b4a80 con 0x7f3cbc15eae0 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.429+0000 7f3cbaffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3cbc0d2360 0x7f3cbc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.429+0000 7f3cbaffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3cbc0d2360 0x7f3cbc0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49460/0 (socket says 172.21.15.12:49460) 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.429+0000 7f3cbaffd640 1 -- 172.21.15.12:0/3528043959 learned_addr learned my addr 172.21.15.12:0/3528043959 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.430+0000 7f3cbaffd640 1 -- 172.21.15.12:0/3528043959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc15eae0 msgr2=0x7f3cbc0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.430+0000 7f3cbaffd640 1 --2- 172.21.15.12:0/3528043959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc15eae0 0x7f3cbc0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.430+0000 7f3cbaffd640 1 -- 172.21.15.12:0/3528043959 >> v1:172.21.15.12:6789/0 conn(0x7f3cbc15e700 legacy=0x7f3cbc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.430+0000 7f3cbaffd640 1 -- 172.21.15.12:0/3528043959 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cbc0b4d60 con 0x7f3cbc0d2360 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.430+0000 7f3cbaffd640 1 --2- 172.21.15.12:0/3528043959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc0d2360 0x7f3cbc0d2b60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009870 tx=0x7f3cb002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbdc4a4147c19bf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cba7fc640 1 -- 172.21.15.12:0/3528043959 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0034070 con 0x7f3cbc0d2360 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cba7fc640 1 -- 172.21.15.12:0/3528043959 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb0030840 con 0x7f3cbc0d2360 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cba7fc640 1 -- 172.21.15.12:0/3528043959 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0030c10 con 0x7f3cbc0d2360 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/3528043959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc0d2360 msgr2=0x7f3cbc0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/3528043959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc0d2360 0x7f3cbc0d2b60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009870 tx=0x7f3cb002b980 comp rx=0 tx=0).stop 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/3528043959 shutdown_connections 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/3528043959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc15eae0 0x7f3cbc0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/3528043959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc0d2360 0x7f3cbc0d2b60 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/3528043959 >> 172.21.15.12:0/3528043959 conn(0x7f3cbc08bf60 msgr2=0x7f3cbc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/3528043959 shutdown_connections 2024-04-15T15:22:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.431+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/3528043959 wait complete. 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.432+0000 7f3cc1bbd640 1 Processor -- start 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cc1bbd640 1 -- start start 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cc1bbd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d2360 0x7f3cbc0cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cc1bbd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e700 0x7f3cbc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cc1bbd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc15eae0 0x7f3cbc15f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cc1bbd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc0d1d40 con 0x7f3cbc15e700 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cc1bbd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc0b4450 con 0x7f3cbc15eae0 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cc1bbd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3cbc0d1ec0 con 0x7f3cbc0d2360 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cbaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e700 0x7f3cbc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cbaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e700 0x7f3cbc15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49468/0 (socket says 172.21.15.12:49468) 2024-04-15T15:22:36.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.433+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2296085960 learned_addr learned my addr 172.21.15.12:0/2296085960 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbb7fe640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d2360 0x7f3cbc0cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d2360 msgr2=0x7f3cbc0cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbaffd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d2360 0x7f3cbc0cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbbfff640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc15eae0 0x7f3cbc15f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc15eae0 msgr2=0x7f3cbc15f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbaffd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc15eae0 0x7f3cbc15f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbaffd640 1 -- 172.21.15.12:0/2296085960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cbc0cdb30 con 0x7f3cbc15e700 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbbfff640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc15eae0 0x7f3cbc15f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cbaffd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e700 0x7f3cbc15ef90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0000c00 tx=0x7f3cb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3ca3fff640 1 -- 172.21.15.12:0/2296085960 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0048070 con 0x7f3cbc15e700 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3ca3fff640 1 -- 172.21.15.12:0/2296085960 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb0030640 con 0x7f3cbc15e700 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3ca3fff640 1 -- 172.21.15.12:0/2296085960 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0034040 con 0x7f3cbc15e700 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cbc0cde10 con 0x7f3cbc15e700 2024-04-15T15:22:36.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.434+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cbc15af90 con 0x7f3cbc15e700 2024-04-15T15:22:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.435+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cbc0689e0 con 0x7f3cbc15e700 2024-04-15T15:22:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.436+0000 7f3ca3fff640 1 -- 172.21.15.12:0/2296085960 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cb0037070 con 0x7f3cbc15e700 2024-04-15T15:22:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.436+0000 7f3ca3fff640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.436+0000 7f3ca3fff640 1 -- 172.21.15.12:0/2296085960 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f3cb00adb50 con 0x7f3cbc15e700 2024-04-15T15:22:36.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.436+0000 7f3cbb7fe640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.437+0000 7f3cbb7fe640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3cac007fd0 tx=0x7f3cac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:36.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.441+0000 7f3ca3fff640 1 -- 172.21.15.12:0/2296085960 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cb007bdf0 con 0x7f3cbc15e700 2024-04-15T15:22:36.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.666+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cbc0ba500 con 0x7f3cbc15e700 2024-04-15T15:22:36.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.666+0000 7f3ca3fff640 1 -- 172.21.15.12:0/2296085960 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7f3cb007b6f0 con 0x7f3cbc15e700 2024-04-15T15:22:36.669 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:36.669 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:22:35.667510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/3388147599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":3388147599},{"type":"v1","addr":"172.21.15.55:6837","nonce":3388147599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5203,"qdb_cluster":[5203]},"id":2}]} 2024-04-15T15:22:36.669 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 18 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 msgr2=0x7f3ca806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3cac007fd0 tx=0x7f3cac009040 comp rx=0 tx=0).stop 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e700 msgr2=0x7f3cbc15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e700 0x7f3cbc15ef90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0000c00 tx=0x7f3cb0004730 comp rx=0 tx=0).stop 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 shutdown_connections 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3ca806c8f0 0x7f3ca806edb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cbc15eae0 0x7f3cbc15f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cbc15e700 0x7f3cbc15ef90 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 --2- 172.21.15.12:0/2296085960 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3cbc0d2360 0x7f3cbc0cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 >> 172.21.15.12:0/2296085960 conn(0x7f3cbc08bf60 msgr2=0x7f3cbc0b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:36.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.668+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 shutdown_connections 2024-04-15T15:22:36.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.669+0000 7f3cc1bbd640 1 -- 172.21.15.12:0/2296085960 wait complete. 2024-04-15T15:22:36.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.785+0000 7fceca1cd640 1 Processor -- start 2024-04-15T15:22:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.785+0000 7fceca1cd640 1 -- start start 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.786+0000 7fceca1cd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcec415e790 0x7fcec40d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.786+0000 7fceca1cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcec40d1610 0x7fcec40d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.786+0000 7fceca1cd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec40d2480 con 0x7fcec40cfd20 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.786+0000 7fceca1cd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcec40d2db0 con 0x7fcec40d1610 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.786+0000 7fceca1cd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fcec40d2760 con 0x7fcec415e790 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.786+0000 7fcec3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcec40d1610 0x7fcec40d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.786+0000 7fcec3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcec40d1610 0x7fcec40d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49472/0 (socket says 172.21.15.12:49472) 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.786+0000 7fcec3fff640 1 -- 172.21.15.12:0/4185970898 learned_addr learned my addr 172.21.15.12:0/4185970898 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.787+0000 7fcec37fe640 1 --2- 172.21.15.12:0/4185970898 >> v2:172.21.15.55:3300/0 conn(0x7fcec415e790 0x7fcec40d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.787+0000 7fcec3fff640 1 -- 172.21.15.12:0/4185970898 >> v2:172.21.15.55:3300/0 conn(0x7fcec415e790 msgr2=0x7fcec40d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.787+0000 7fcec3fff640 1 --2- 172.21.15.12:0/4185970898 >> v2:172.21.15.55:3300/0 conn(0x7fcec415e790 0x7fcec40d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.787+0000 7fcec3fff640 1 -- 172.21.15.12:0/4185970898 >> v1:172.21.15.12:6789/0 conn(0x7fcec40cfd20 legacy=0x7fcec40d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.787+0000 7fcec3fff640 1 -- 172.21.15.12:0/4185970898 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec40d2a40 con 0x7fcec40d1610 2024-04-15T15:22:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.787+0000 7fcec3fff640 1 --2- 172.21.15.12:0/4185970898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40d1610 0x7fcec40d0160 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009e50 tx=0x7fcebc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33730460bab91d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fcec2ffd640 1 -- 172.21.15.12:0/4185970898 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc02cb20 con 0x7fcec40d1610 2024-04-15T15:22:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fcec2ffd640 1 -- 172.21.15.12:0/4185970898 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcebc02ccc0 con 0x7fcec40d1610 2024-04-15T15:22:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fcec2ffd640 1 -- 172.21.15.12:0/4185970898 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc034610 con 0x7fcec40d1610 2024-04-15T15:22:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fceca1cd640 1 -- 172.21.15.12:0/4185970898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40d1610 msgr2=0x7fcec40d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fceca1cd640 1 --2- 172.21.15.12:0/4185970898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40d1610 0x7fcec40d0160 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009e50 tx=0x7fcebc02bf80 comp rx=0 tx=0).stop 2024-04-15T15:22:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fceca1cd640 1 -- 172.21.15.12:0/4185970898 shutdown_connections 2024-04-15T15:22:36.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fceca1cd640 1 --2- 172.21.15.12:0/4185970898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40d1610 0x7fcec40d0160 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fceca1cd640 1 --2- 172.21.15.12:0/4185970898 >> v2:172.21.15.55:3300/0 conn(0x7fcec415e790 0x7fcec40d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fceca1cd640 1 -- 172.21.15.12:0/4185970898 >> 172.21.15.12:0/4185970898 conn(0x7fcec408bf60 msgr2=0x7fcec40d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fceca1cd640 1 -- 172.21.15.12:0/4185970898 shutdown_connections 2024-04-15T15:22:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.788+0000 7fceca1cd640 1 -- 172.21.15.12:0/4185970898 wait complete. 2024-04-15T15:22:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.789+0000 7fceca1cd640 1 Processor -- start 2024-04-15T15:22:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fceca1cd640 1 -- start start 2024-04-15T15:22:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fceca1cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40cfd20 0x7fcec4152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fceca1cd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcec40d1610 0x7fcec40ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fceca1cd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec415e790 0x7fcec4151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fceca1cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcec415ecf0 con 0x7fcec40cfd20 2024-04-15T15:22:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fceca1cd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcec415eb70 con 0x7fcec415e790 2024-04-15T15:22:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fceca1cd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcec40b74f0 con 0x7fcec40d1610 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40cfd20 0x7fcec4152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40cfd20 0x7fcec4152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49476/0 (socket says 172.21.15.12:49476) 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.790+0000 7fcec37fe640 1 -- 172.21.15.12:0/2131672621 learned_addr learned my addr 172.21.15.12:0/2131672621 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec3fff640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec415e790 0x7fcec4151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcebaffd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcec40d1610 0x7fcec40ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec37fe640 1 -- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcec40d1610 msgr2=0x7fcec40ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec37fe640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcec40d1610 0x7fcec40ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec37fe640 1 -- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec415e790 msgr2=0x7fcec4151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec37fe640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec415e790 0x7fcec4151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec37fe640 1 -- 172.21.15.12:0/2131672621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec4153300 con 0x7fcec40cfd20 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec3fff640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec415e790 0x7fcec4151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec37fe640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40cfd20 0x7fcec4152ab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fceb400d620 tx=0x7fceb400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec0ff9640 1 -- 172.21.15.12:0/2131672621 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb400b620 con 0x7fcec40cfd20 2024-04-15T15:22:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec0ff9640 1 -- 172.21.15.12:0/2131672621 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb4019070 con 0x7fcec40cfd20 2024-04-15T15:22:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec41535e0 con 0x7fcec40cfd20 2024-04-15T15:22:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fcec0ff9640 1 -- 172.21.15.12:0/2131672621 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb400ba00 con 0x7fcec40cfd20 2024-04-15T15:22:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.791+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec4153ba0 con 0x7fcec40cfd20 2024-04-15T15:22:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.792+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec40689e0 con 0x7fcec40cfd20 2024-04-15T15:22:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.793+0000 7fcec0ff9640 1 -- 172.21.15.12:0/2131672621 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fceb400bba0 con 0x7fcec40cfd20 2024-04-15T15:22:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.793+0000 7fcec0ff9640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcea406c8f0 0x7fcea406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.793+0000 7fcec0ff9640 1 -- 172.21.15.12:0/2131672621 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fceb408c0d0 con 0x7fcec40cfd20 2024-04-15T15:22:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.793+0000 7fcebaffd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcea406c8f0 0x7fcea406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.794+0000 7fcebaffd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcea406c8f0 0x7fcea406edb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcec40b4c10 tx=0x7fceac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.798+0000 7fcec0ff9640 1 -- 172.21.15.12:0/2131672621 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fceb405a370 con 0x7fcec40cfd20 2024-04-15T15:22:36.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.981+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcec40d1ce0 con 0x7fcec40cfd20 2024-04-15T15:22:36.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.982+0000 7fcec0ff9640 1 -- 172.21.15.12:0/2131672621 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16906 (secure 0 0 0) 0x7fceb4060030 con 0x7fcec40cfd20 2024-04-15T15:22:36.984 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:36.984 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":39,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:22:34.287725+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:36.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcea406c8f0 msgr2=0x7fcea406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcea406c8f0 0x7fcea406edb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcec40b4c10 tx=0x7fceac009040 comp rx=0 tx=0).stop 2024-04-15T15:22:36.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40cfd20 msgr2=0x7fcec4152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40cfd20 0x7fcec4152ab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fceb400d620 tx=0x7fceb400dae0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 shutdown_connections 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcea406c8f0 0x7fcea406edb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec415e790 0x7fcec4151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcec40d1610 0x7fcec40ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 --2- 172.21.15.12:0/2131672621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcec40cfd20 0x7fcec4152ab0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 >> 172.21.15.12:0/2131672621 conn(0x7fcec408bf60 msgr2=0x7fcec40b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 shutdown_connections 2024-04-15T15:22:36.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:36.984+0000 7fceca1cd640 1 -- 172.21.15.12:0/2131672621 wait complete. 2024-04-15T15:22:36.997 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-15T15:22:36.997 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:22:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.099+0000 7fbfde8c5640 1 Processor -- start 2024-04-15T15:22:37.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.099+0000 7fbfde8c5640 1 -- start start 2024-04-15T15:22:37.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.100+0000 7fbfde8c5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd815e790 0x7fbfd80b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.100+0000 7fbfde8c5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbfd80d1cf0 0x7fbfd80d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.100+0000 7fbfde8c5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd815eb70 con 0x7fbfd80b7be0 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.100+0000 7fbfde8c5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fbfd80d3790 con 0x7fbfd80d1cf0 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.100+0000 7fbfde8c5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd80d45b0 con 0x7fbfd815e790 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.100+0000 7fbfcffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbfd80b7be0 0x7fbfd80d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35940/0 (socket says 172.21.15.12:35940) 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.100+0000 7fbfcffff640 1 -- 172.21.15.12:0/879041249 learned_addr learned my addr 172.21.15.12:0/879041249 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd7fff640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd815e790 0x7fbfd80b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfdce62640 1 --2- 172.21.15.12:0/879041249 >> v2:172.21.15.55:3300/0 conn(0x7fbfd80d1cf0 0x7fbfd80d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33054672 0 0) 0x7fbfd815eb70 con 0x7fbfd80b7be0 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfc4003520 con 0x7fbfd80b7be0 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3644769578 0 0) 0x7fbfc4003520 con 0x7fbfd80b7be0 2024-04-15T15:22:37.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd815e790 msgr2=0x7fbfd80b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd815e790 0x7fbfd80b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 >> v2:172.21.15.55:3300/0 conn(0x7fbfd80d1cf0 msgr2=0x7fbfd80d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 --2- 172.21.15.12:0/879041249 >> v2:172.21.15.55:3300/0 conn(0x7fbfd80d1cf0 0x7fbfd80d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd80d3c80 con 0x7fbfd80b7be0 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fbfc0001690 con 0x7fbfd80b7be0 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 >> v1:172.21.15.12:6789/0 conn(0x7fbfd80b7be0 legacy=0x7fbfd80d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc40039b0 0x7fbfc4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd815e790 0x7fbfc4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfc4004d40 0x7fbfc40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80d3790 con 0x7fbfc4004d40 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80d45b0 con 0x7fbfc40039b0 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc400ba70 con 0x7fbfd815e790 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfdce62640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfc4004d40 0x7fbfc40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfd7fff640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc40039b0 0x7fbfc4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfcffff640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd815e790 0x7fbfc4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfdce62640 1 -- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd815e790 msgr2=0x7fbfc4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfdce62640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd815e790 0x7fbfc4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfdce62640 1 -- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc40039b0 msgr2=0x7fbfc4003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfdce62640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc40039b0 0x7fbfc4003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.101+0000 7fbfdce62640 1 -- 172.21.15.12:0/879041249 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc4003520 con 0x7fbfc4004d40 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.102+0000 7fbfdce62640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfc4004d40 0x7fbfc40051c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80d2460 tx=0x7fbfc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06f8c9a55351a74 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.102+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0039070 con 0x7fbfc4004d40 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.102+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd002cbd0 con 0x7fbfc4004d40 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.102+0000 7fbfd77fe640 1 -- 172.21.15.12:0/879041249 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0009c10 con 0x7fbfc4004d40 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.103+0000 7fbfde8c5640 1 -- 172.21.15.12:0/879041249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfc4004d40 msgr2=0x7fbfc40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.103+0000 7fbfde8c5640 1 --2- 172.21.15.12:0/879041249 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfc4004d40 0x7fbfc40051c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80d2460 tx=0x7fbfc4005830 comp rx=0 tx=0).stop 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.103+0000 7fbfdce62640 1 -- 172.21.15.12:0/879041249 reap_dead start 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.103+0000 7fbfde8c5640 1 -- 172.21.15.12:0/879041249 shutdown_connections 2024-04-15T15:22:37.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.103+0000 7fbfde8c5640 1 -- 172.21.15.12:0/879041249 >> 172.21.15.12:0/879041249 conn(0x7fbfd8067b40 msgr2=0x7fbfd8067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.103+0000 7fbfde8c5640 1 -- 172.21.15.12:0/879041249 shutdown_connections 2024-04-15T15:22:37.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.103+0000 7fbfde8c5640 1 -- 172.21.15.12:0/879041249 wait complete. 2024-04-15T15:22:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.104+0000 7fbfde8c5640 1 Processor -- start 2024-04-15T15:22:37.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.104+0000 7fbfde8c5640 1 -- start start 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfde8c5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc4004d40 0x7fbfd815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfde8c5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd80c77d0 0x7fbfd80c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfde8c5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd80c8660 0x7fbfd80c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfde8c5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80b76a0 con 0x7fbfd80c77d0 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfde8c5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80d4240 con 0x7fbfc4004d40 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfde8c5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd80b7820 con 0x7fbfd80c8660 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd80c77d0 0x7fbfd80c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd80c77d0 0x7fbfd80c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49494/0 (socket says 172.21.15.12:49494) 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 -- 172.21.15.12:0/3806427635 learned_addr learned my addr 172.21.15.12:0/3806427635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfdce62640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd80c8660 0x7fbfd80c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfd7fff640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc4004d40 0x7fbfd815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 -- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd80c8660 msgr2=0x7fbfd80c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd80c8660 0x7fbfd80c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 -- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc4004d40 msgr2=0x7fbfd815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc4004d40 0x7fbfd815c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 -- 172.21.15.12:0/3806427635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd81f4660 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.105+0000 7fbfcffff640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd80c77d0 0x7fbfd80c8120 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000e700 tx=0x7fbfc000eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.106+0000 7fbfd57fa640 1 -- 172.21.15.12:0/3806427635 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc000c620 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.106+0000 7fbfd57fa640 1 -- 172.21.15.12:0/3806427635 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc0018070 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.106+0000 7fbfd57fa640 1 -- 172.21.15.12:0/3806427635 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc000ca00 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.106+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd80ab000 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.106+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd80ab2c0 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.107+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd80689e0 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.107+0000 7fbfd57fa640 1 -- 172.21.15.12:0/3806427635 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfc000cba0 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.107+0000 7fbfd57fa640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbfb806c920 0x7fbfb806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.108+0000 7fbfd57fa640 1 -- 172.21.15.12:0/3806427635 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbfc008b910 con 0x7fbfd80c77d0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.108+0000 7fbfd7fff640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbfb806c920 0x7fbfb806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.108+0000 7fbfd7fff640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbfb806c920 0x7fbfb806ede0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009bc0 tx=0x7fbfc8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.112+0000 7fbfd57fa640 1 -- 172.21.15.12:0/3806427635 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbfc0059c40 con 0x7fbfd80c77d0 2024-04-15T15:22:37.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.295+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbfd815e040 con 0x7fbfb806c920 2024-04-15T15:22:37.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.302+0000 7fbfd57fa640 1 -- 172.21.15.12:0/3806427635 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7fbfd815e040 con 0x7fbfb806c920 2024-04-15T15:22:37.307 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:37.308 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:22:37.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbfb806c920 msgr2=0x7fbfb806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbfb806c920 0x7fbfb806ede0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009bc0 tx=0x7fbfc8009190 comp rx=0 tx=0).stop 2024-04-15T15:22:37.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd80c77d0 msgr2=0x7fbfd80c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd80c77d0 0x7fbfd80c8120 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000e700 tx=0x7fbfc000eb00 comp rx=0 tx=0).stop 2024-04-15T15:22:37.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 shutdown_connections 2024-04-15T15:22:37.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fbfd80c8660 0x7fbfd80c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd80c77d0 0x7fbfd80c8120 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fbfb806c920 0x7fbfb806ede0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 --2- 172.21.15.12:0/3806427635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfc4004d40 0x7fbfd815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 >> 172.21.15.12:0/3806427635 conn(0x7fbfd8067b40 msgr2=0x7fbfd8161fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:37.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 shutdown_connections 2024-04-15T15:22:37.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.305+0000 7fbfde8c5640 1 -- 172.21.15.12:0/3806427635 wait complete. 2024-04-15T15:22:37.319 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-15T15:22:36.608251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237216,"kb_used_data":4416,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737504,"statfs":{"total":773094113280,"available":772851204096,"internally_reserved":0,"allocated":4521984,"data_stored":2590110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.331612"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283803+0000","last_change":"2024-04-15T15:22:32.283803+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283803+0000","last_undegraded":"2024-04-15T15:22:32.283803+0000","last_fullsized":"2024-04-15T15:22:32.283803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282846+0000","last_change":"2024-04-15T15:22:32.282846+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282846+0000","last_undegraded":"2024-04-15T15:22:32.282846+0000","last_fullsized":"2024-04-15T15:22:32.282846+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282557+0000","last_change":"2024-04-15T15:22:32.282557+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282557+0000","last_undegraded":"2024-04-15T15:22:32.282557+0000","last_fullsized":"2024-04-15T15:22:32.282557+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285184+0000","last_change":"2024-04-15T15:22:32.285184+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285184+0000","last_undegraded":"2024-04-15T15:22:32.285184+0000","last_fullsized":"2024-04-15T15:22:32.285184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285654+0000","last_change":"2024-04-15T15:22:32.285654+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285654+0000","last_undegraded":"2024-04-15T15:22:32.285654+0000","last_fullsized":"2024-04-15T15:22:32.285654+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288873+0000","last_change":"2024-04-15T15:22:32.288873+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288873+0000","last_undegraded":"2024-04-15T15:22:32.288873+0000","last_fullsized":"2024-04-15T15:22:32.288873+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.289028+0000","last_change":"2024-04-15T15:22:32.289028+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.289028+0000","last_undegraded":"2024-04-15T15:22:32.289028+0000","last_fullsized":"2024-04-15T15:22:32.289028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281732+0000","last_change":"2024-04-15T15:22:32.281732+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281732+0000","last_undegraded":"2024-04-15T15:22:32.281732+0000","last_fullsized":"2024-04-15T15:22:32.281732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282504+0000","last_change":"2024-04-15T15:22:32.282504+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282504+0000","last_undegraded":"2024-04-15T15:22:32.282504+0000","last_fullsized":"2024-04-15T15:22:32.282504+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284678+0000","last_change":"2024-04-15T15:22:32.284678+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284678+0000","last_undegraded":"2024-04-15T15:22:32.284678+0000","last_fullsized":"2024-04-15T15:22:32.284678+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286253+0000","last_change":"2024-04-15T15:22:32.286253+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286253+0000","last_undegraded":"2024-04-15T15:22:32.286253+0000","last_fullsized":"2024-04-15T15:22:32.286253+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282565+0000","last_change":"2024-04-15T15:22:32.282565+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282565+0000","last_undegraded":"2024-04-15T15:22:32.282565+0000","last_fullsized":"2024-04-15T15:22:32.282565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.289197+0000","last_change":"2024-04-15T15:22:32.289197+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.289197+0000","last_undegraded":"2024-04-15T15:22:32.289197+0000","last_fullsized":"2024-04-15T15:22:32.289197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286855+0000","last_change":"2024-04-15T15:22:32.286855+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286855+0000","last_undegraded":"2024-04-15T15:22:32.286855+0000","last_fullsized":"2024-04-15T15:22:32.286855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288550+0000","last_change":"2024-04-15T15:22:32.288550+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288550+0000","last_undegraded":"2024-04-15T15:22:32.288550+0000","last_fullsized":"2024-04-15T15:22:32.288550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287794+0000","last_change":"2024-04-15T15:22:32.287794+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287794+0000","last_undegraded":"2024-04-15T15:22:32.287794+0000","last_fullsized":"2024-04-15T15:22:32.287794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287169+0000","last_change":"2024-04-15T15:22:32.287169+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287169+0000","last_undegraded":"2024-04-15T15:22:32.287169+0000","last_fullsized":"2024-04-15T15:22:32.287169+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283134+0000","last_change":"2024-04-15T15:22:32.283134+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283134+0000","last_undegraded":"2024-04-15T15:22:32.283134+0000","last_fullsized":"2024-04-15T15:22:32.283134+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287033+0000","last_change":"2024-04-15T15:22:32.287033+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287033+0000","last_undegraded":"2024-04-15T15:22:32.287033+0000","last_fullsized":"2024-04-15T15:22:32.287033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283165+0000","last_change":"2024-04-15T15:22:32.283165+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283165+0000","last_undegraded":"2024-04-15T15:22:32.283165+0000","last_fullsized":"2024-04-15T15:22:32.283165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287661+0000","last_change":"2024-04-15T15:22:32.287661+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287661+0000","last_undegraded":"2024-04-15T15:22:32.287661+0000","last_fullsized":"2024-04-15T15:22:32.287661+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282573+0000","last_change":"2024-04-15T15:22:32.282573+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282573+0000","last_undegraded":"2024-04-15T15:22:32.282573+0000","last_fullsized":"2024-04-15T15:22:32.282573+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288692+0000","last_change":"2024-04-15T15:22:32.288692+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288692+0000","last_undegraded":"2024-04-15T15:22:32.288692+0000","last_fullsized":"2024-04-15T15:22:32.288692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286380+0000","last_change":"2024-04-15T15:22:32.286380+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286380+0000","last_undegraded":"2024-04-15T15:22:32.286380+0000","last_fullsized":"2024-04-15T15:22:32.286380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283560+0000","last_change":"2024-04-15T15:22:32.283560+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283560+0000","last_undegraded":"2024-04-15T15:22:32.283560+0000","last_fullsized":"2024-04-15T15:22:32.283560+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286540+0000","last_change":"2024-04-15T15:22:32.286540+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286540+0000","last_undegraded":"2024-04-15T15:22:32.286540+0000","last_fullsized":"2024-04-15T15:22:32.286540+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282295+0000","last_change":"2024-04-15T15:22:32.282295+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282295+0000","last_undegraded":"2024-04-15T15:22:32.282295+0000","last_fullsized":"2024-04-15T15:22:32.282295+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287436+0000","last_change":"2024-04-15T15:22:32.287436+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287436+0000","last_undegraded":"2024-04-15T15:22:32.287436+0000","last_fullsized":"2024-04-15T15:22:32.287436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284239+0000","last_change":"2024-04-15T15:22:32.284239+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284239+0000","last_undegraded":"2024-04-15T15:22:32.284239+0000","last_fullsized":"2024-04-15T15:22:32.284239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283378+0000","last_change":"2024-04-15T15:22:32.283378+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283378+0000","last_undegraded":"2024-04-15T15:22:32.283378+0000","last_fullsized":"2024-04-15T15:22:32.283378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286533+0000","last_change":"2024-04-15T15:22:32.286533+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286533+0000","last_undegraded":"2024-04-15T15:22:32.286533+0000","last_fullsized":"2024-04-15T15:22:32.286533+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286417+0000","last_change":"2024-04-15T15:22:32.286417+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286417+0000","last_undegraded":"2024-04-15T15:22:32.286417+0000","last_fullsized":"2024-04-15T15:22:32.286417+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287816+0000","last_change":"2024-04-15T15:22:32.287816+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287816+0000","last_undegraded":"2024-04-15T15:22:32.287816+0000","last_fullsized":"2024-04-15T15:22:32.287816+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300082+0000","last_change":"2024-04-15T15:22:32.295369+0000","last_active":"2024-04-15T15:22:32.300082+0000","last_peered":"2024-04-15T15:22:32.300082+0000","last_clean":"2024-04-15T15:22:32.300082+0000","last_became_active":"2024-04-15T15:22:32.295054+0000","last_became_peered":"2024-04-15T15:22:32.295054+0000","last_unstale":"2024-04-15T15:22:32.300082+0000","last_undegraded":"2024-04-15T15:22:32.300082+0000","last_fullsized":"2024-04-15T15:22:32.300082+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:05:16.011118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288081+0000","last_change":"2024-04-15T15:22:32.288081+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288081+0000","last_undegraded":"2024-04-15T15:22:32.288081+0000","last_fullsized":"2024-04-15T15:22:32.288081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303013+0000","last_change":"2024-04-15T15:22:32.298663+0000","last_active":"2024-04-15T15:22:32.303013+0000","last_peered":"2024-04-15T15:22:32.303013+0000","last_clean":"2024-04-15T15:22:32.303013+0000","last_became_active":"2024-04-15T15:22:32.298296+0000","last_became_peered":"2024-04-15T15:22:32.298296+0000","last_unstale":"2024-04-15T15:22:32.303013+0000","last_undegraded":"2024-04-15T15:22:32.303013+0000","last_fullsized":"2024-04-15T15:22:32.303013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284349+0000","last_change":"2024-04-15T15:22:32.284349+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284349+0000","last_undegraded":"2024-04-15T15:22:32.284349+0000","last_fullsized":"2024-04-15T15:22:32.284349+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306108+0000","last_change":"2024-04-15T15:22:32.301621+0000","last_active":"2024-04-15T15:22:32.306108+0000","last_peered":"2024-04-15T15:22:32.306108+0000","last_clean":"2024-04-15T15:22:32.306108+0000","last_became_active":"2024-04-15T15:22:32.301253+0000","last_became_peered":"2024-04-15T15:22:32.301253+0000","last_unstale":"2024-04-15T15:22:32.306108+0000","last_undegraded":"2024-04-15T15:22:32.306108+0000","last_fullsized":"2024-04-15T15:22:32.306108+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283316+0000","last_change":"2024-04-15T15:22:32.283316+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283316+0000","last_undegraded":"2024-04-15T15:22:32.283316+0000","last_fullsized":"2024-04-15T15:22:32.283316+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301630+0000","last_change":"2024-04-15T15:22:32.297657+0000","last_active":"2024-04-15T15:22:32.301630+0000","last_peered":"2024-04-15T15:22:32.301630+0000","last_clean":"2024-04-15T15:22:32.301630+0000","last_became_active":"2024-04-15T15:22:32.297419+0000","last_became_peered":"2024-04-15T15:22:32.297419+0000","last_unstale":"2024-04-15T15:22:32.301630+0000","last_undegraded":"2024-04-15T15:22:32.301630+0000","last_fullsized":"2024-04-15T15:22:32.301630+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:09:33.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282144+0000","last_change":"2024-04-15T15:22:32.282144+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282144+0000","last_undegraded":"2024-04-15T15:22:32.282144+0000","last_fullsized":"2024-04-15T15:22:32.282144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300355+0000","last_change":"2024-04-15T15:22:32.293808+0000","last_active":"2024-04-15T15:22:32.300355+0000","last_peered":"2024-04-15T15:22:32.300355+0000","last_clean":"2024-04-15T15:22:32.300355+0000","last_became_active":"2024-04-15T15:22:32.293474+0000","last_became_peered":"2024-04-15T15:22:32.293474+0000","last_unstale":"2024-04-15T15:22:32.300355+0000","last_undegraded":"2024-04-15T15:22:32.300355+0000","last_fullsized":"2024-04-15T15:22:32.300355+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287768+0000","last_change":"2024-04-15T15:22:32.287768+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287768+0000","last_undegraded":"2024-04-15T15:22:32.287768+0000","last_fullsized":"2024-04-15T15:22:32.287768+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301034+0000","last_change":"2024-04-15T15:22:32.299916+0000","last_active":"2024-04-15T15:22:32.301034+0000","last_peered":"2024-04-15T15:22:32.301034+0000","last_clean":"2024-04-15T15:22:32.301034+0000","last_became_active":"2024-04-15T15:22:32.299576+0000","last_became_peered":"2024-04-15T15:22:32.299576+0000","last_unstale":"2024-04-15T15:22:32.301034+0000","last_undegraded":"2024-04-15T15:22:32.301034+0000","last_fullsized":"2024-04-15T15:22:32.301034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282000+0000","last_change":"2024-04-15T15:22:32.282000+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282000+0000","last_undegraded":"2024-04-15T15:22:32.282000+0000","last_fullsized":"2024-04-15T15:22:32.282000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306074+0000","last_change":"2024-04-15T15:22:32.301232+0000","last_active":"2024-04-15T15:22:32.306074+0000","last_peered":"2024-04-15T15:22:32.306074+0000","last_clean":"2024-04-15T15:22:32.306074+0000","last_became_active":"2024-04-15T15:22:32.300885+0000","last_became_peered":"2024-04-15T15:22:32.300885+0000","last_unstale":"2024-04-15T15:22:32.306074+0000","last_undegraded":"2024-04-15T15:22:32.306074+0000","last_fullsized":"2024-04-15T15:22:32.306074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:01.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282531+0000","last_change":"2024-04-15T15:22:32.282531+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282531+0000","last_undegraded":"2024-04-15T15:22:32.282531+0000","last_fullsized":"2024-04-15T15:22:32.282531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293532+0000","last_change":"2024-04-15T15:22:32.288888+0000","last_active":"2024-04-15T15:22:32.293532+0000","last_peered":"2024-04-15T15:22:32.293532+0000","last_clean":"2024-04-15T15:22:32.293532+0000","last_became_active":"2024-04-15T15:22:32.288559+0000","last_became_peered":"2024-04-15T15:22:32.288559+0000","last_unstale":"2024-04-15T15:22:32.293532+0000","last_undegraded":"2024-04-15T15:22:32.293532+0000","last_fullsized":"2024-04-15T15:22:32.293532+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:02:52.088900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284589+0000","last_change":"2024-04-15T15:22:32.284589+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284589+0000","last_undegraded":"2024-04-15T15:22:32.284589+0000","last_fullsized":"2024-04-15T15:22:32.284589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293989+0000","last_change":"2024-04-15T15:22:32.292353+0000","last_active":"2024-04-15T15:22:32.293989+0000","last_peered":"2024-04-15T15:22:32.293989+0000","last_clean":"2024-04-15T15:22:32.293989+0000","last_became_active":"2024-04-15T15:22:32.292043+0000","last_became_peered":"2024-04-15T15:22:32.292043+0000","last_unstale":"2024-04-15T15:22:32.293989+0000","last_undegraded":"2024-04-15T15:22:32.293989+0000","last_fullsized":"2024-04-15T15:22:32.293989+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:21:12.998104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288577+0000","last_change":"2024-04-15T15:22:32.288577+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288577+0000","last_undegraded":"2024-04-15T15:22:32.288577+0000","last_fullsized":"2024-04-15T15:22:32.288577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301133+0000","last_change":"2024-04-15T15:22:32.294706+0000","last_active":"2024-04-15T15:22:32.301133+0000","last_peered":"2024-04-15T15:22:32.301133+0000","last_clean":"2024-04-15T15:22:32.301133+0000","last_became_active":"2024-04-15T15:22:32.294353+0000","last_became_peered":"2024-04-15T15:22:32.294353+0000","last_unstale":"2024-04-15T15:22:32.301133+0000","last_undegraded":"2024-04-15T15:22:32.301133+0000","last_fullsized":"2024-04-15T15:22:32.301133+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281997+0000","last_change":"2024-04-15T15:22:32.281997+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281997+0000","last_undegraded":"2024-04-15T15:22:32.281997+0000","last_fullsized":"2024-04-15T15:22:32.281997+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301078+0000","last_change":"2024-04-15T15:22:32.294321+0000","last_active":"2024-04-15T15:22:32.301078+0000","last_peered":"2024-04-15T15:22:32.301078+0000","last_clean":"2024-04-15T15:22:32.301078+0000","last_became_active":"2024-04-15T15:22:32.294043+0000","last_became_peered":"2024-04-15T15:22:32.294043+0000","last_unstale":"2024-04-15T15:22:32.301078+0000","last_undegraded":"2024-04-15T15:22:32.301078+0000","last_fullsized":"2024-04-15T15:22:32.301078+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284953+0000","last_change":"2024-04-15T15:22:32.284953+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284953+0000","last_undegraded":"2024-04-15T15:22:32.284953+0000","last_fullsized":"2024-04-15T15:22:32.284953+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.305720+0000","last_change":"2024-04-15T15:22:32.298894+0000","last_active":"2024-04-15T15:22:32.305720+0000","last_peered":"2024-04-15T15:22:32.305720+0000","last_clean":"2024-04-15T15:22:32.305720+0000","last_became_active":"2024-04-15T15:22:32.298534+0000","last_became_peered":"2024-04-15T15:22:32.298534+0000","last_unstale":"2024-04-15T15:22:32.305720+0000","last_undegraded":"2024-04-15T15:22:32.305720+0000","last_fullsized":"2024-04-15T15:22:32.305720+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:45.706167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281544+0000","last_change":"2024-04-15T15:22:32.281544+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281544+0000","last_undegraded":"2024-04-15T15:22:32.281544+0000","last_fullsized":"2024-04-15T15:22:32.281544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.292179+0000","last_change":"2024-04-15T15:22:32.287177+0000","last_active":"2024-04-15T15:22:32.292179+0000","last_peered":"2024-04-15T15:22:32.292179+0000","last_clean":"2024-04-15T15:22:32.292179+0000","last_became_active":"2024-04-15T15:22:32.286887+0000","last_became_peered":"2024-04-15T15:22:32.286887+0000","last_unstale":"2024-04-15T15:22:32.292179+0000","last_undegraded":"2024-04-15T15:22:32.292179+0000","last_fullsized":"2024-04-15T15:22:32.292179+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281381+0000","last_change":"2024-04-15T15:22:32.281381+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281381+0000","last_undegraded":"2024-04-15T15:22:32.281381+0000","last_fullsized":"2024-04-15T15:22:32.281381+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306066+0000","last_change":"2024-04-15T15:22:32.301053+0000","last_active":"2024-04-15T15:22:32.306066+0000","last_peered":"2024-04-15T15:22:32.306066+0000","last_clean":"2024-04-15T15:22:32.306066+0000","last_became_active":"2024-04-15T15:22:32.300755+0000","last_became_peered":"2024-04-15T15:22:32.300755+0000","last_unstale":"2024-04-15T15:22:32.306066+0000","last_undegraded":"2024-04-15T15:22:32.306066+0000","last_fullsized":"2024-04-15T15:22:32.306066+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:09:33.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285849+0000","last_change":"2024-04-15T15:22:32.285849+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285849+0000","last_undegraded":"2024-04-15T15:22:32.285849+0000","last_fullsized":"2024-04-15T15:22:32.285849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.295745+0000","last_change":"2024-04-15T15:22:32.293921+0000","last_active":"2024-04-15T15:22:32.295745+0000","last_peered":"2024-04-15T15:22:32.295745+0000","last_clean":"2024-04-15T15:22:32.295745+0000","last_became_active":"2024-04-15T15:22:32.293641+0000","last_became_peered":"2024-04-15T15:22:32.293641+0000","last_unstale":"2024-04-15T15:22:32.295745+0000","last_undegraded":"2024-04-15T15:22:32.295745+0000","last_fullsized":"2024-04-15T15:22:32.295745+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:02:52.088900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282090+0000","last_change":"2024-04-15T15:22:32.282090+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282090+0000","last_undegraded":"2024-04-15T15:22:32.282090+0000","last_fullsized":"2024-04-15T15:22:32.282090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301077+0000","last_change":"2024-04-15T15:22:32.299339+0000","last_active":"2024-04-15T15:22:32.301077+0000","last_peered":"2024-04-15T15:22:32.301077+0000","last_clean":"2024-04-15T15:22:32.301077+0000","last_became_active":"2024-04-15T15:22:32.298921+0000","last_became_peered":"2024-04-15T15:22:32.298921+0000","last_unstale":"2024-04-15T15:22:32.301077+0000","last_undegraded":"2024-04-15T15:22:32.301077+0000","last_fullsized":"2024-04-15T15:22:32.301077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281983+0000","last_change":"2024-04-15T15:22:32.281983+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281983+0000","last_undegraded":"2024-04-15T15:22:32.281983+0000","last_fullsized":"2024-04-15T15:22:32.281983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301166+0000","last_change":"2024-04-15T15:22:32.299157+0000","last_active":"2024-04-15T15:22:32.301166+0000","last_peered":"2024-04-15T15:22:32.301166+0000","last_clean":"2024-04-15T15:22:32.301166+0000","last_became_active":"2024-04-15T15:22:32.298836+0000","last_became_peered":"2024-04-15T15:22:32.298836+0000","last_unstale":"2024-04-15T15:22:32.301166+0000","last_undegraded":"2024-04-15T15:22:32.301166+0000","last_fullsized":"2024-04-15T15:22:32.301166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285787+0000","last_change":"2024-04-15T15:22:32.285787+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285787+0000","last_undegraded":"2024-04-15T15:22:32.285787+0000","last_fullsized":"2024-04-15T15:22:32.285787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.302895+0000","last_change":"2024-04-15T15:22:32.301976+0000","last_active":"2024-04-15T15:22:32.302895+0000","last_peered":"2024-04-15T15:22:32.302895+0000","last_clean":"2024-04-15T15:22:32.302895+0000","last_became_active":"2024-04-15T15:22:32.301692+0000","last_became_peered":"2024-04-15T15:22:32.301692+0000","last_unstale":"2024-04-15T15:22:32.302895+0000","last_undegraded":"2024-04-15T15:22:32.302895+0000","last_fullsized":"2024-04-15T15:22:32.302895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:01.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284873+0000","last_change":"2024-04-15T15:22:32.284873+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284873+0000","last_undegraded":"2024-04-15T15:22:32.284873+0000","last_fullsized":"2024-04-15T15:22:32.284873+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303068+0000","last_change":"2024-04-15T15:22:32.299415+0000","last_active":"2024-04-15T15:22:32.303068+0000","last_peered":"2024-04-15T15:22:32.303068+0000","last_clean":"2024-04-15T15:22:32.303068+0000","last_became_active":"2024-04-15T15:22:32.299078+0000","last_became_peered":"2024-04-15T15:22:32.299078+0000","last_unstale":"2024-04-15T15:22:32.303068+0000","last_undegraded":"2024-04-15T15:22:32.303068+0000","last_fullsized":"2024-04-15T15:22:32.303068+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:17:24.243206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285847+0000","last_change":"2024-04-15T15:22:32.285847+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285847+0000","last_undegraded":"2024-04-15T15:22:32.285847+0000","last_fullsized":"2024-04-15T15:22:32.285847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306094+0000","last_change":"2024-04-15T15:22:32.301462+0000","last_active":"2024-04-15T15:22:32.306094+0000","last_peered":"2024-04-15T15:22:32.306094+0000","last_clean":"2024-04-15T15:22:32.306094+0000","last_became_active":"2024-04-15T15:22:32.301176+0000","last_became_peered":"2024-04-15T15:22:32.301176+0000","last_unstale":"2024-04-15T15:22:32.306094+0000","last_undegraded":"2024-04-15T15:22:32.306094+0000","last_fullsized":"2024-04-15T15:22:32.306094+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285710+0000","last_change":"2024-04-15T15:22:32.285710+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285710+0000","last_undegraded":"2024-04-15T15:22:32.285710+0000","last_fullsized":"2024-04-15T15:22:32.285710+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293378+0000","last_change":"2024-04-15T15:22:32.290430+0000","last_active":"2024-04-15T15:22:32.293378+0000","last_peered":"2024-04-15T15:22:32.293378+0000","last_clean":"2024-04-15T15:22:32.293378+0000","last_became_active":"2024-04-15T15:22:32.290116+0000","last_became_peered":"2024-04-15T15:22:32.290116+0000","last_unstale":"2024-04-15T15:22:32.293378+0000","last_undegraded":"2024-04-15T15:22:32.293378+0000","last_fullsized":"2024-04-15T15:22:32.293378+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:28:10.589579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.280827+0000","last_change":"2024-04-15T15:22:32.280827+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.280827+0000","last_undegraded":"2024-04-15T15:22:32.280827+0000","last_fullsized":"2024-04-15T15:22:32.280827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303060+0000","last_change":"2024-04-15T15:22:32.299094+0000","last_active":"2024-04-15T15:22:32.303060+0000","last_peered":"2024-04-15T15:22:32.303060+0000","last_clean":"2024-04-15T15:22:32.303060+0000","last_became_active":"2024-04-15T15:22:32.298792+0000","last_became_peered":"2024-04-15T15:22:32.298792+0000","last_unstale":"2024-04-15T15:22:32.303060+0000","last_undegraded":"2024-04-15T15:22:32.303060+0000","last_fullsized":"2024-04-15T15:22:32.303060+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:46:18.965814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284940+0000","last_change":"2024-04-15T15:22:32.284940+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284940+0000","last_undegraded":"2024-04-15T15:22:32.284940+0000","last_fullsized":"2024-04-15T15:22:32.284940+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301061+0000","last_change":"2024-04-15T15:22:32.298790+0000","last_active":"2024-04-15T15:22:32.301061+0000","last_peered":"2024-04-15T15:22:32.301061+0000","last_clean":"2024-04-15T15:22:32.301061+0000","last_became_active":"2024-04-15T15:22:32.298475+0000","last_became_peered":"2024-04-15T15:22:32.298475+0000","last_unstale":"2024-04-15T15:22:32.301061+0000","last_undegraded":"2024-04-15T15:22:32.301061+0000","last_fullsized":"2024-04-15T15:22:32.301061+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281646+0000","last_change":"2024-04-15T15:22:32.281646+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281646+0000","last_undegraded":"2024-04-15T15:22:32.281646+0000","last_fullsized":"2024-04-15T15:22:32.281646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.294638+0000","last_change":"2024-04-15T15:22:32.293012+0000","last_active":"2024-04-15T15:22:32.294638+0000","last_peered":"2024-04-15T15:22:32.294638+0000","last_clean":"2024-04-15T15:22:32.294638+0000","last_became_active":"2024-04-15T15:22:32.292728+0000","last_became_peered":"2024-04-15T15:22:32.292728+0000","last_unstale":"2024-04-15T15:22:32.294638+0000","last_undegraded":"2024-04-15T15:22:32.294638+0000","last_fullsized":"2024-04-15T15:22:32.294638+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:44.182373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285229+0000","last_change":"2024-04-15T15:22:32.285229+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285229+0000","last_undegraded":"2024-04-15T15:22:32.285229+0000","last_fullsized":"2024-04-15T15:22:32.285229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301126+0000","last_change":"2024-04-15T15:22:32.295146+0000","last_active":"2024-04-15T15:22:32.301126+0000","last_peered":"2024-04-15T15:22:32.301126+0000","last_clean":"2024-04-15T15:22:32.301126+0000","last_became_active":"2024-04-15T15:22:32.294842+0000","last_became_peered":"2024-04-15T15:22:32.294842+0000","last_unstale":"2024-04-15T15:22:32.301126+0000","last_undegraded":"2024-04-15T15:22:32.301126+0000","last_fullsized":"2024-04-15T15:22:32.301126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288112+0000","last_change":"2024-04-15T15:22:32.288112+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288112+0000","last_undegraded":"2024-04-15T15:22:32.288112+0000","last_fullsized":"2024-04-15T15:22:32.288112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.296985+0000","last_change":"2024-04-15T15:22:32.295490+0000","last_active":"2024-04-15T15:22:32.296985+0000","last_peered":"2024-04-15T15:22:32.296985+0000","last_clean":"2024-04-15T15:22:32.296985+0000","last_became_active":"2024-04-15T15:22:32.295129+0000","last_became_peered":"2024-04-15T15:22:32.295129+0000","last_unstale":"2024-04-15T15:22:32.296985+0000","last_undegraded":"2024-04-15T15:22:32.296985+0000","last_fullsized":"2024-04-15T15:22:32.296985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:26:37.527479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282394+0000","last_change":"2024-04-15T15:22:32.282394+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282394+0000","last_undegraded":"2024-04-15T15:22:32.282394+0000","last_fullsized":"2024-04-15T15:22:32.282394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.295606+0000","last_change":"2024-04-15T15:22:32.294246+0000","last_active":"2024-04-15T15:22:32.295606+0000","last_peered":"2024-04-15T15:22:32.295606+0000","last_clean":"2024-04-15T15:22:32.295606+0000","last_became_active":"2024-04-15T15:22:32.293952+0000","last_became_peered":"2024-04-15T15:22:32.293952+0000","last_unstale":"2024-04-15T15:22:32.295606+0000","last_undegraded":"2024-04-15T15:22:32.295606+0000","last_fullsized":"2024-04-15T15:22:32.295606+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.280827+0000","last_change":"2024-04-15T15:22:32.280827+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.280827+0000","last_undegraded":"2024-04-15T15:22:32.280827+0000","last_fullsized":"2024-04-15T15:22:32.280827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.292565+0000","last_change":"2024-04-15T15:22:32.288288+0000","last_active":"2024-04-15T15:22:32.292565+0000","last_peered":"2024-04-15T15:22:32.292565+0000","last_clean":"2024-04-15T15:22:32.292565+0000","last_became_active":"2024-04-15T15:22:32.287969+0000","last_became_peered":"2024-04-15T15:22:32.287969+0000","last_unstale":"2024-04-15T15:22:32.292565+0000","last_undegraded":"2024-04-15T15:22:32.292565+0000","last_fullsized":"2024-04-15T15:22:32.292565+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286818+0000","last_change":"2024-04-15T15:22:32.286818+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286818+0000","last_undegraded":"2024-04-15T15:22:32.286818+0000","last_fullsized":"2024-04-15T15:22:32.286818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.294372+0000","last_change":"2024-04-15T15:22:32.292570+0000","last_active":"2024-04-15T15:22:32.294372+0000","last_peered":"2024-04-15T15:22:32.294372+0000","last_clean":"2024-04-15T15:22:32.294372+0000","last_became_active":"2024-04-15T15:22:32.292275+0000","last_became_peered":"2024-04-15T15:22:32.292275+0000","last_unstale":"2024-04-15T15:22:32.294372+0000","last_undegraded":"2024-04-15T15:22:32.294372+0000","last_fullsized":"2024-04-15T15:22:32.294372+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:10:15.989931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283018+0000","last_change":"2024-04-15T15:22:32.283018+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283018+0000","last_undegraded":"2024-04-15T15:22:32.283018+0000","last_fullsized":"2024-04-15T15:22:32.283018+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301534+0000","last_change":"2024-04-15T15:22:32.297615+0000","last_active":"2024-04-15T15:22:32.301534+0000","last_peered":"2024-04-15T15:22:32.301534+0000","last_clean":"2024-04-15T15:22:32.301534+0000","last_became_active":"2024-04-15T15:22:32.297385+0000","last_became_peered":"2024-04-15T15:22:32.297385+0000","last_unstale":"2024-04-15T15:22:32.301534+0000","last_undegraded":"2024-04-15T15:22:32.301534+0000","last_fullsized":"2024-04-15T15:22:32.301534+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:05:16.011118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287505+0000","last_change":"2024-04-15T15:22:32.287505+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287505+0000","last_undegraded":"2024-04-15T15:22:32.287505+0000","last_fullsized":"2024-04-15T15:22:32.287505+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293952+0000","last_change":"2024-04-15T15:22:32.292214+0000","last_active":"2024-04-15T15:22:32.293952+0000","last_peered":"2024-04-15T15:22:32.293952+0000","last_clean":"2024-04-15T15:22:32.293952+0000","last_became_active":"2024-04-15T15:22:32.291892+0000","last_became_peered":"2024-04-15T15:22:32.291892+0000","last_unstale":"2024-04-15T15:22:32.293952+0000","last_undegraded":"2024-04-15T15:22:32.293952+0000","last_fullsized":"2024-04-15T15:22:32.293952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285960+0000","last_change":"2024-04-15T15:22:32.285960+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285960+0000","last_undegraded":"2024-04-15T15:22:32.285960+0000","last_fullsized":"2024-04-15T15:22:32.285960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300939+0000","last_change":"2024-04-15T15:22:32.299060+0000","last_active":"2024-04-15T15:22:32.300939+0000","last_peered":"2024-04-15T15:22:32.300939+0000","last_clean":"2024-04-15T15:22:32.300939+0000","last_became_active":"2024-04-15T15:22:32.298599+0000","last_became_peered":"2024-04-15T15:22:32.298599+0000","last_unstale":"2024-04-15T15:22:32.300939+0000","last_undegraded":"2024-04-15T15:22:32.300939+0000","last_fullsized":"2024-04-15T15:22:32.300939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.289893+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:22:32.289893+0000","last_peered":"2024-04-15T15:22:32.289893+0000","last_clean":"2024-04-15T15:22:32.289893+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:22:32.289893+0000","last_undegraded":"2024-04-15T15:22:32.289893+0000","last_fullsized":"2024-04-15T15:22:32.289893+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":771540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":764873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:22:37.324 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:22:37.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.461+0000 7f361c2eb640 1 Processor -- start 2024-04-15T15:22:37.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.461+0000 7f361c2eb640 1 -- start start 2024-04-15T15:22:37.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.462+0000 7f361c2eb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f361415e700 0x7f36140b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.462+0000 7f361c2eb640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f36140d2360 0x7f36140d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.462+0000 7f361c2eb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36140b47a0 con 0x7f361415eae0 2024-04-15T15:22:37.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.462+0000 7f361c2eb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f36140b53b0 con 0x7f361415e700 2024-04-15T15:22:37.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.462+0000 7f361c2eb640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f36140b4a80 con 0x7f36140d2360 2024-04-15T15:22:37.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.462+0000 7f361a087640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f361415e700 0x7f36140b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.462+0000 7f361a087640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f361415e700 0x7f36140b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49504/0 (socket says 172.21.15.12:49504) 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.462+0000 7f361a087640 1 -- 172.21.15.12:0/2606730971 learned_addr learned my addr 172.21.15.12:0/2606730971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.463+0000 7f3619886640 1 --2- 172.21.15.12:0/2606730971 >> v2:172.21.15.55:3300/0 conn(0x7f36140d2360 0x7f36140d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.463+0000 7f361a087640 1 -- 172.21.15.12:0/2606730971 >> v2:172.21.15.55:3300/0 conn(0x7f36140d2360 msgr2=0x7f36140d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.463+0000 7f361a087640 1 --2- 172.21.15.12:0/2606730971 >> v2:172.21.15.55:3300/0 conn(0x7f36140d2360 0x7f36140d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.463+0000 7f361a087640 1 -- 172.21.15.12:0/2606730971 >> v1:172.21.15.12:6789/0 conn(0x7f361415eae0 legacy=0x7f36140cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.463+0000 7f361a087640 1 -- 172.21.15.12:0/2606730971 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36140b4d60 con 0x7f361415e700 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.463+0000 7f361a087640 1 --2- 172.21.15.12:0/2606730971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b5840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f361408c7b0 tx=0x7f360402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5480cc9a5829b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f3619085640 1 -- 172.21.15.12:0/2606730971 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360402cb70 con 0x7f361415e700 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f3619085640 1 -- 172.21.15.12:0/2606730971 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f360402cd10 con 0x7f361415e700 2024-04-15T15:22:37.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f3619085640 1 -- 172.21.15.12:0/2606730971 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604034610 con 0x7f361415e700 2024-04-15T15:22:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f361c2eb640 1 -- 172.21.15.12:0/2606730971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 msgr2=0x7f36140b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f361c2eb640 1 --2- 172.21.15.12:0/2606730971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b5840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f361408c7b0 tx=0x7f360402bfd0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f361c2eb640 1 -- 172.21.15.12:0/2606730971 shutdown_connections 2024-04-15T15:22:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f361c2eb640 1 --2- 172.21.15.12:0/2606730971 >> v2:172.21.15.55:3300/0 conn(0x7f36140d2360 0x7f36140d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f361c2eb640 1 --2- 172.21.15.12:0/2606730971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b5840 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f361c2eb640 1 -- 172.21.15.12:0/2606730971 >> 172.21.15.12:0/2606730971 conn(0x7f361408bf60 msgr2=0x7f36140d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f361c2eb640 1 -- 172.21.15.12:0/2606730971 shutdown_connections 2024-04-15T15:22:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.464+0000 7f361c2eb640 1 -- 172.21.15.12:0/2606730971 wait complete. 2024-04-15T15:22:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.465+0000 7f361c2eb640 1 Processor -- start 2024-04-15T15:22:37.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f361c2eb640 1 -- start start 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f361c2eb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36140d2360 0x7f36140ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f361c2eb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f361c2eb640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f361415eae0 0x7f36140b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f361c2eb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f36140b5d80 con 0x7f361415e700 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f361c2eb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f36140b4450 con 0x7f36140d2360 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f361c2eb640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f36140d1d40 con 0x7f361415eae0 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f3619886640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f3619886640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49508/0 (socket says 172.21.15.12:49508) 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.466+0000 7f3619886640 1 -- 172.21.15.12:0/1518305677 learned_addr learned my addr 172.21.15.12:0/1518305677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3619886640 1 -- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f361415eae0 msgr2=0x7f36140b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3619886640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f361415eae0 0x7f36140b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3619886640 1 -- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36140d2360 msgr2=0x7f36140ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3619886640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36140d2360 0x7f36140ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3619886640 1 -- 172.21.15.12:0/1518305677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36140b95f0 con 0x7f361415e700 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3619886640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b37e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f360800e9b0 tx=0x7f360800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3612ffd640 1 -- 172.21.15.12:0/1518305677 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608017070 con 0x7f361415e700 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3612ffd640 1 -- 172.21.15.12:0/1518305677 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f360800c9a0 con 0x7f361415e700 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f3612ffd640 1 -- 172.21.15.12:0/1518305677 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360800cd90 con 0x7f361415e700 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36140b98d0 con 0x7f361415e700 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.467+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36140b9e90 con 0x7f361415e700 2024-04-15T15:22:37.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.468+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36140689e0 con 0x7f361415e700 2024-04-15T15:22:37.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.468+0000 7f3612ffd640 1 -- 172.21.15.12:0/1518305677 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f360801a070 con 0x7f361415e700 2024-04-15T15:22:37.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.469+0000 7f3612ffd640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f360006c920 0x7f360006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.469+0000 7f3612ffd640 1 -- 172.21.15.12:0/1518305677 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f360808c070 con 0x7f361415e700 2024-04-15T15:22:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.469+0000 7f361a087640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f360006c920 0x7f360006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.470+0000 7f361a087640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f360006c920 0x7f360006ede0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3604035000 tx=0x7f360402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.473+0000 7f3612ffd640 1 -- 172.21.15.12:0/1518305677 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f360805a310 con 0x7f361415e700 2024-04-15T15:22:37.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.654+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f36140ba500 con 0x7f360006c920 2024-04-15T15:22:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.660+0000 7f3612ffd640 1 -- 172.21.15.12:0/1518305677 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7f36140ba500 con 0x7f360006c920 2024-04-15T15:22:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:37.667 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:22:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f360006c920 msgr2=0x7f360006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f360006c920 0x7f360006ede0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3604035000 tx=0x7f360402e000 comp rx=0 tx=0).stop 2024-04-15T15:22:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 msgr2=0x7f36140b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b37e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f360800e9b0 tx=0x7f360800ee70 comp rx=0 tx=0).stop 2024-04-15T15:22:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 shutdown_connections 2024-04-15T15:22:37.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f361415eae0 0x7f36140b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f361415e700 0x7f36140b37e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f360006c920 0x7f360006ede0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.663+0000 7f361c2eb640 1 --2- 172.21.15.12:0/1518305677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36140d2360 0x7f36140ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.664+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 >> 172.21.15.12:0/1518305677 conn(0x7f361408bf60 msgr2=0x7f36140cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.664+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 shutdown_connections 2024-04-15T15:22:37.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.664+0000 7f361c2eb640 1 -- 172.21.15.12:0/1518305677 wait complete. 2024-04-15T15:22:37.677 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-15T15:22:36.608251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237216,"kb_used_data":4416,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737504,"statfs":{"total":773094113280,"available":772851204096,"internally_reserved":0,"allocated":4521984,"data_stored":2590110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.331612"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283803+0000","last_change":"2024-04-15T15:22:32.283803+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283803+0000","last_undegraded":"2024-04-15T15:22:32.283803+0000","last_fullsized":"2024-04-15T15:22:32.283803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282846+0000","last_change":"2024-04-15T15:22:32.282846+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282846+0000","last_undegraded":"2024-04-15T15:22:32.282846+0000","last_fullsized":"2024-04-15T15:22:32.282846+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282557+0000","last_change":"2024-04-15T15:22:32.282557+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282557+0000","last_undegraded":"2024-04-15T15:22:32.282557+0000","last_fullsized":"2024-04-15T15:22:32.282557+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285184+0000","last_change":"2024-04-15T15:22:32.285184+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285184+0000","last_undegraded":"2024-04-15T15:22:32.285184+0000","last_fullsized":"2024-04-15T15:22:32.285184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285654+0000","last_change":"2024-04-15T15:22:32.285654+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285654+0000","last_undegraded":"2024-04-15T15:22:32.285654+0000","last_fullsized":"2024-04-15T15:22:32.285654+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288873+0000","last_change":"2024-04-15T15:22:32.288873+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288873+0000","last_undegraded":"2024-04-15T15:22:32.288873+0000","last_fullsized":"2024-04-15T15:22:32.288873+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.289028+0000","last_change":"2024-04-15T15:22:32.289028+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.289028+0000","last_undegraded":"2024-04-15T15:22:32.289028+0000","last_fullsized":"2024-04-15T15:22:32.289028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281732+0000","last_change":"2024-04-15T15:22:32.281732+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281732+0000","last_undegraded":"2024-04-15T15:22:32.281732+0000","last_fullsized":"2024-04-15T15:22:32.281732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282504+0000","last_change":"2024-04-15T15:22:32.282504+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282504+0000","last_undegraded":"2024-04-15T15:22:32.282504+0000","last_fullsized":"2024-04-15T15:22:32.282504+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284678+0000","last_change":"2024-04-15T15:22:32.284678+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284678+0000","last_undegraded":"2024-04-15T15:22:32.284678+0000","last_fullsized":"2024-04-15T15:22:32.284678+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286253+0000","last_change":"2024-04-15T15:22:32.286253+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286253+0000","last_undegraded":"2024-04-15T15:22:32.286253+0000","last_fullsized":"2024-04-15T15:22:32.286253+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282565+0000","last_change":"2024-04-15T15:22:32.282565+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282565+0000","last_undegraded":"2024-04-15T15:22:32.282565+0000","last_fullsized":"2024-04-15T15:22:32.282565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.289197+0000","last_change":"2024-04-15T15:22:32.289197+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.289197+0000","last_undegraded":"2024-04-15T15:22:32.289197+0000","last_fullsized":"2024-04-15T15:22:32.289197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286855+0000","last_change":"2024-04-15T15:22:32.286855+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286855+0000","last_undegraded":"2024-04-15T15:22:32.286855+0000","last_fullsized":"2024-04-15T15:22:32.286855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288550+0000","last_change":"2024-04-15T15:22:32.288550+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288550+0000","last_undegraded":"2024-04-15T15:22:32.288550+0000","last_fullsized":"2024-04-15T15:22:32.288550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287794+0000","last_change":"2024-04-15T15:22:32.287794+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287794+0000","last_undegraded":"2024-04-15T15:22:32.287794+0000","last_fullsized":"2024-04-15T15:22:32.287794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287169+0000","last_change":"2024-04-15T15:22:32.287169+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287169+0000","last_undegraded":"2024-04-15T15:22:32.287169+0000","last_fullsized":"2024-04-15T15:22:32.287169+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283134+0000","last_change":"2024-04-15T15:22:32.283134+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283134+0000","last_undegraded":"2024-04-15T15:22:32.283134+0000","last_fullsized":"2024-04-15T15:22:32.283134+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287033+0000","last_change":"2024-04-15T15:22:32.287033+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287033+0000","last_undegraded":"2024-04-15T15:22:32.287033+0000","last_fullsized":"2024-04-15T15:22:32.287033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283165+0000","last_change":"2024-04-15T15:22:32.283165+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283165+0000","last_undegraded":"2024-04-15T15:22:32.283165+0000","last_fullsized":"2024-04-15T15:22:32.283165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287661+0000","last_change":"2024-04-15T15:22:32.287661+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287661+0000","last_undegraded":"2024-04-15T15:22:32.287661+0000","last_fullsized":"2024-04-15T15:22:32.287661+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282573+0000","last_change":"2024-04-15T15:22:32.282573+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282573+0000","last_undegraded":"2024-04-15T15:22:32.282573+0000","last_fullsized":"2024-04-15T15:22:32.282573+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288692+0000","last_change":"2024-04-15T15:22:32.288692+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288692+0000","last_undegraded":"2024-04-15T15:22:32.288692+0000","last_fullsized":"2024-04-15T15:22:32.288692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286380+0000","last_change":"2024-04-15T15:22:32.286380+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286380+0000","last_undegraded":"2024-04-15T15:22:32.286380+0000","last_fullsized":"2024-04-15T15:22:32.286380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283560+0000","last_change":"2024-04-15T15:22:32.283560+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283560+0000","last_undegraded":"2024-04-15T15:22:32.283560+0000","last_fullsized":"2024-04-15T15:22:32.283560+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286540+0000","last_change":"2024-04-15T15:22:32.286540+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286540+0000","last_undegraded":"2024-04-15T15:22:32.286540+0000","last_fullsized":"2024-04-15T15:22:32.286540+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282295+0000","last_change":"2024-04-15T15:22:32.282295+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282295+0000","last_undegraded":"2024-04-15T15:22:32.282295+0000","last_fullsized":"2024-04-15T15:22:32.282295+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287436+0000","last_change":"2024-04-15T15:22:32.287436+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287436+0000","last_undegraded":"2024-04-15T15:22:32.287436+0000","last_fullsized":"2024-04-15T15:22:32.287436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284239+0000","last_change":"2024-04-15T15:22:32.284239+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284239+0000","last_undegraded":"2024-04-15T15:22:32.284239+0000","last_fullsized":"2024-04-15T15:22:32.284239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283378+0000","last_change":"2024-04-15T15:22:32.283378+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283378+0000","last_undegraded":"2024-04-15T15:22:32.283378+0000","last_fullsized":"2024-04-15T15:22:32.283378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286533+0000","last_change":"2024-04-15T15:22:32.286533+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286533+0000","last_undegraded":"2024-04-15T15:22:32.286533+0000","last_fullsized":"2024-04-15T15:22:32.286533+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286417+0000","last_change":"2024-04-15T15:22:32.286417+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286417+0000","last_undegraded":"2024-04-15T15:22:32.286417+0000","last_fullsized":"2024-04-15T15:22:32.286417+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287816+0000","last_change":"2024-04-15T15:22:32.287816+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287816+0000","last_undegraded":"2024-04-15T15:22:32.287816+0000","last_fullsized":"2024-04-15T15:22:32.287816+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300082+0000","last_change":"2024-04-15T15:22:32.295369+0000","last_active":"2024-04-15T15:22:32.300082+0000","last_peered":"2024-04-15T15:22:32.300082+0000","last_clean":"2024-04-15T15:22:32.300082+0000","last_became_active":"2024-04-15T15:22:32.295054+0000","last_became_peered":"2024-04-15T15:22:32.295054+0000","last_unstale":"2024-04-15T15:22:32.300082+0000","last_undegraded":"2024-04-15T15:22:32.300082+0000","last_fullsized":"2024-04-15T15:22:32.300082+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:05:16.011118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288081+0000","last_change":"2024-04-15T15:22:32.288081+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288081+0000","last_undegraded":"2024-04-15T15:22:32.288081+0000","last_fullsized":"2024-04-15T15:22:32.288081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303013+0000","last_change":"2024-04-15T15:22:32.298663+0000","last_active":"2024-04-15T15:22:32.303013+0000","last_peered":"2024-04-15T15:22:32.303013+0000","last_clean":"2024-04-15T15:22:32.303013+0000","last_became_active":"2024-04-15T15:22:32.298296+0000","last_became_peered":"2024-04-15T15:22:32.298296+0000","last_unstale":"2024-04-15T15:22:32.303013+0000","last_undegraded":"2024-04-15T15:22:32.303013+0000","last_fullsized":"2024-04-15T15:22:32.303013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284349+0000","last_change":"2024-04-15T15:22:32.284349+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284349+0000","last_undegraded":"2024-04-15T15:22:32.284349+0000","last_fullsized":"2024-04-15T15:22:32.284349+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306108+0000","last_change":"2024-04-15T15:22:32.301621+0000","last_active":"2024-04-15T15:22:32.306108+0000","last_peered":"2024-04-15T15:22:32.306108+0000","last_clean":"2024-04-15T15:22:32.306108+0000","last_became_active":"2024-04-15T15:22:32.301253+0000","last_became_peered":"2024-04-15T15:22:32.301253+0000","last_unstale":"2024-04-15T15:22:32.306108+0000","last_undegraded":"2024-04-15T15:22:32.306108+0000","last_fullsized":"2024-04-15T15:22:32.306108+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283316+0000","last_change":"2024-04-15T15:22:32.283316+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283316+0000","last_undegraded":"2024-04-15T15:22:32.283316+0000","last_fullsized":"2024-04-15T15:22:32.283316+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301630+0000","last_change":"2024-04-15T15:22:32.297657+0000","last_active":"2024-04-15T15:22:32.301630+0000","last_peered":"2024-04-15T15:22:32.301630+0000","last_clean":"2024-04-15T15:22:32.301630+0000","last_became_active":"2024-04-15T15:22:32.297419+0000","last_became_peered":"2024-04-15T15:22:32.297419+0000","last_unstale":"2024-04-15T15:22:32.301630+0000","last_undegraded":"2024-04-15T15:22:32.301630+0000","last_fullsized":"2024-04-15T15:22:32.301630+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:09:33.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282144+0000","last_change":"2024-04-15T15:22:32.282144+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282144+0000","last_undegraded":"2024-04-15T15:22:32.282144+0000","last_fullsized":"2024-04-15T15:22:32.282144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300355+0000","last_change":"2024-04-15T15:22:32.293808+0000","last_active":"2024-04-15T15:22:32.300355+0000","last_peered":"2024-04-15T15:22:32.300355+0000","last_clean":"2024-04-15T15:22:32.300355+0000","last_became_active":"2024-04-15T15:22:32.293474+0000","last_became_peered":"2024-04-15T15:22:32.293474+0000","last_unstale":"2024-04-15T15:22:32.300355+0000","last_undegraded":"2024-04-15T15:22:32.300355+0000","last_fullsized":"2024-04-15T15:22:32.300355+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287768+0000","last_change":"2024-04-15T15:22:32.287768+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287768+0000","last_undegraded":"2024-04-15T15:22:32.287768+0000","last_fullsized":"2024-04-15T15:22:32.287768+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301034+0000","last_change":"2024-04-15T15:22:32.299916+0000","last_active":"2024-04-15T15:22:32.301034+0000","last_peered":"2024-04-15T15:22:32.301034+0000","last_clean":"2024-04-15T15:22:32.301034+0000","last_became_active":"2024-04-15T15:22:32.299576+0000","last_became_peered":"2024-04-15T15:22:32.299576+0000","last_unstale":"2024-04-15T15:22:32.301034+0000","last_undegraded":"2024-04-15T15:22:32.301034+0000","last_fullsized":"2024-04-15T15:22:32.301034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282000+0000","last_change":"2024-04-15T15:22:32.282000+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282000+0000","last_undegraded":"2024-04-15T15:22:32.282000+0000","last_fullsized":"2024-04-15T15:22:32.282000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306074+0000","last_change":"2024-04-15T15:22:32.301232+0000","last_active":"2024-04-15T15:22:32.306074+0000","last_peered":"2024-04-15T15:22:32.306074+0000","last_clean":"2024-04-15T15:22:32.306074+0000","last_became_active":"2024-04-15T15:22:32.300885+0000","last_became_peered":"2024-04-15T15:22:32.300885+0000","last_unstale":"2024-04-15T15:22:32.306074+0000","last_undegraded":"2024-04-15T15:22:32.306074+0000","last_fullsized":"2024-04-15T15:22:32.306074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:01.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282531+0000","last_change":"2024-04-15T15:22:32.282531+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282531+0000","last_undegraded":"2024-04-15T15:22:32.282531+0000","last_fullsized":"2024-04-15T15:22:32.282531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293532+0000","last_change":"2024-04-15T15:22:32.288888+0000","last_active":"2024-04-15T15:22:32.293532+0000","last_peered":"2024-04-15T15:22:32.293532+0000","last_clean":"2024-04-15T15:22:32.293532+0000","last_became_active":"2024-04-15T15:22:32.288559+0000","last_became_peered":"2024-04-15T15:22:32.288559+0000","last_unstale":"2024-04-15T15:22:32.293532+0000","last_undegraded":"2024-04-15T15:22:32.293532+0000","last_fullsized":"2024-04-15T15:22:32.293532+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:02:52.088900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284589+0000","last_change":"2024-04-15T15:22:32.284589+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284589+0000","last_undegraded":"2024-04-15T15:22:32.284589+0000","last_fullsized":"2024-04-15T15:22:32.284589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293989+0000","last_change":"2024-04-15T15:22:32.292353+0000","last_active":"2024-04-15T15:22:32.293989+0000","last_peered":"2024-04-15T15:22:32.293989+0000","last_clean":"2024-04-15T15:22:32.293989+0000","last_became_active":"2024-04-15T15:22:32.292043+0000","last_became_peered":"2024-04-15T15:22:32.292043+0000","last_unstale":"2024-04-15T15:22:32.293989+0000","last_undegraded":"2024-04-15T15:22:32.293989+0000","last_fullsized":"2024-04-15T15:22:32.293989+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:21:12.998104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288577+0000","last_change":"2024-04-15T15:22:32.288577+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288577+0000","last_undegraded":"2024-04-15T15:22:32.288577+0000","last_fullsized":"2024-04-15T15:22:32.288577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301133+0000","last_change":"2024-04-15T15:22:32.294706+0000","last_active":"2024-04-15T15:22:32.301133+0000","last_peered":"2024-04-15T15:22:32.301133+0000","last_clean":"2024-04-15T15:22:32.301133+0000","last_became_active":"2024-04-15T15:22:32.294353+0000","last_became_peered":"2024-04-15T15:22:32.294353+0000","last_unstale":"2024-04-15T15:22:32.301133+0000","last_undegraded":"2024-04-15T15:22:32.301133+0000","last_fullsized":"2024-04-15T15:22:32.301133+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281997+0000","last_change":"2024-04-15T15:22:32.281997+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281997+0000","last_undegraded":"2024-04-15T15:22:32.281997+0000","last_fullsized":"2024-04-15T15:22:32.281997+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301078+0000","last_change":"2024-04-15T15:22:32.294321+0000","last_active":"2024-04-15T15:22:32.301078+0000","last_peered":"2024-04-15T15:22:32.301078+0000","last_clean":"2024-04-15T15:22:32.301078+0000","last_became_active":"2024-04-15T15:22:32.294043+0000","last_became_peered":"2024-04-15T15:22:32.294043+0000","last_unstale":"2024-04-15T15:22:32.301078+0000","last_undegraded":"2024-04-15T15:22:32.301078+0000","last_fullsized":"2024-04-15T15:22:32.301078+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284953+0000","last_change":"2024-04-15T15:22:32.284953+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284953+0000","last_undegraded":"2024-04-15T15:22:32.284953+0000","last_fullsized":"2024-04-15T15:22:32.284953+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.305720+0000","last_change":"2024-04-15T15:22:32.298894+0000","last_active":"2024-04-15T15:22:32.305720+0000","last_peered":"2024-04-15T15:22:32.305720+0000","last_clean":"2024-04-15T15:22:32.305720+0000","last_became_active":"2024-04-15T15:22:32.298534+0000","last_became_peered":"2024-04-15T15:22:32.298534+0000","last_unstale":"2024-04-15T15:22:32.305720+0000","last_undegraded":"2024-04-15T15:22:32.305720+0000","last_fullsized":"2024-04-15T15:22:32.305720+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:45.706167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281544+0000","last_change":"2024-04-15T15:22:32.281544+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281544+0000","last_undegraded":"2024-04-15T15:22:32.281544+0000","last_fullsized":"2024-04-15T15:22:32.281544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.292179+0000","last_change":"2024-04-15T15:22:32.287177+0000","last_active":"2024-04-15T15:22:32.292179+0000","last_peered":"2024-04-15T15:22:32.292179+0000","last_clean":"2024-04-15T15:22:32.292179+0000","last_became_active":"2024-04-15T15:22:32.286887+0000","last_became_peered":"2024-04-15T15:22:32.286887+0000","last_unstale":"2024-04-15T15:22:32.292179+0000","last_undegraded":"2024-04-15T15:22:32.292179+0000","last_fullsized":"2024-04-15T15:22:32.292179+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281381+0000","last_change":"2024-04-15T15:22:32.281381+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281381+0000","last_undegraded":"2024-04-15T15:22:32.281381+0000","last_fullsized":"2024-04-15T15:22:32.281381+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306066+0000","last_change":"2024-04-15T15:22:32.301053+0000","last_active":"2024-04-15T15:22:32.306066+0000","last_peered":"2024-04-15T15:22:32.306066+0000","last_clean":"2024-04-15T15:22:32.306066+0000","last_became_active":"2024-04-15T15:22:32.300755+0000","last_became_peered":"2024-04-15T15:22:32.300755+0000","last_unstale":"2024-04-15T15:22:32.306066+0000","last_undegraded":"2024-04-15T15:22:32.306066+0000","last_fullsized":"2024-04-15T15:22:32.306066+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:09:33.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285849+0000","last_change":"2024-04-15T15:22:32.285849+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285849+0000","last_undegraded":"2024-04-15T15:22:32.285849+0000","last_fullsized":"2024-04-15T15:22:32.285849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.295745+0000","last_change":"2024-04-15T15:22:32.293921+0000","last_active":"2024-04-15T15:22:32.295745+0000","last_peered":"2024-04-15T15:22:32.295745+0000","last_clean":"2024-04-15T15:22:32.295745+0000","last_became_active":"2024-04-15T15:22:32.293641+0000","last_became_peered":"2024-04-15T15:22:32.293641+0000","last_unstale":"2024-04-15T15:22:32.295745+0000","last_undegraded":"2024-04-15T15:22:32.295745+0000","last_fullsized":"2024-04-15T15:22:32.295745+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:02:52.088900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282090+0000","last_change":"2024-04-15T15:22:32.282090+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282090+0000","last_undegraded":"2024-04-15T15:22:32.282090+0000","last_fullsized":"2024-04-15T15:22:32.282090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301077+0000","last_change":"2024-04-15T15:22:32.299339+0000","last_active":"2024-04-15T15:22:32.301077+0000","last_peered":"2024-04-15T15:22:32.301077+0000","last_clean":"2024-04-15T15:22:32.301077+0000","last_became_active":"2024-04-15T15:22:32.298921+0000","last_became_peered":"2024-04-15T15:22:32.298921+0000","last_unstale":"2024-04-15T15:22:32.301077+0000","last_undegraded":"2024-04-15T15:22:32.301077+0000","last_fullsized":"2024-04-15T15:22:32.301077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281983+0000","last_change":"2024-04-15T15:22:32.281983+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281983+0000","last_undegraded":"2024-04-15T15:22:32.281983+0000","last_fullsized":"2024-04-15T15:22:32.281983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301166+0000","last_change":"2024-04-15T15:22:32.299157+0000","last_active":"2024-04-15T15:22:32.301166+0000","last_peered":"2024-04-15T15:22:32.301166+0000","last_clean":"2024-04-15T15:22:32.301166+0000","last_became_active":"2024-04-15T15:22:32.298836+0000","last_became_peered":"2024-04-15T15:22:32.298836+0000","last_unstale":"2024-04-15T15:22:32.301166+0000","last_undegraded":"2024-04-15T15:22:32.301166+0000","last_fullsized":"2024-04-15T15:22:32.301166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285787+0000","last_change":"2024-04-15T15:22:32.285787+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285787+0000","last_undegraded":"2024-04-15T15:22:32.285787+0000","last_fullsized":"2024-04-15T15:22:32.285787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.302895+0000","last_change":"2024-04-15T15:22:32.301976+0000","last_active":"2024-04-15T15:22:32.302895+0000","last_peered":"2024-04-15T15:22:32.302895+0000","last_clean":"2024-04-15T15:22:32.302895+0000","last_became_active":"2024-04-15T15:22:32.301692+0000","last_became_peered":"2024-04-15T15:22:32.301692+0000","last_unstale":"2024-04-15T15:22:32.302895+0000","last_undegraded":"2024-04-15T15:22:32.302895+0000","last_fullsized":"2024-04-15T15:22:32.302895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:01.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284873+0000","last_change":"2024-04-15T15:22:32.284873+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284873+0000","last_undegraded":"2024-04-15T15:22:32.284873+0000","last_fullsized":"2024-04-15T15:22:32.284873+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303068+0000","last_change":"2024-04-15T15:22:32.299415+0000","last_active":"2024-04-15T15:22:32.303068+0000","last_peered":"2024-04-15T15:22:32.303068+0000","last_clean":"2024-04-15T15:22:32.303068+0000","last_became_active":"2024-04-15T15:22:32.299078+0000","last_became_peered":"2024-04-15T15:22:32.299078+0000","last_unstale":"2024-04-15T15:22:32.303068+0000","last_undegraded":"2024-04-15T15:22:32.303068+0000","last_fullsized":"2024-04-15T15:22:32.303068+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:17:24.243206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285847+0000","last_change":"2024-04-15T15:22:32.285847+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285847+0000","last_undegraded":"2024-04-15T15:22:32.285847+0000","last_fullsized":"2024-04-15T15:22:32.285847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306094+0000","last_change":"2024-04-15T15:22:32.301462+0000","last_active":"2024-04-15T15:22:32.306094+0000","last_peered":"2024-04-15T15:22:32.306094+0000","last_clean":"2024-04-15T15:22:32.306094+0000","last_became_active":"2024-04-15T15:22:32.301176+0000","last_became_peered":"2024-04-15T15:22:32.301176+0000","last_unstale":"2024-04-15T15:22:32.306094+0000","last_undegraded":"2024-04-15T15:22:32.306094+0000","last_fullsized":"2024-04-15T15:22:32.306094+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285710+0000","last_change":"2024-04-15T15:22:32.285710+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285710+0000","last_undegraded":"2024-04-15T15:22:32.285710+0000","last_fullsized":"2024-04-15T15:22:32.285710+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293378+0000","last_change":"2024-04-15T15:22:32.290430+0000","last_active":"2024-04-15T15:22:32.293378+0000","last_peered":"2024-04-15T15:22:32.293378+0000","last_clean":"2024-04-15T15:22:32.293378+0000","last_became_active":"2024-04-15T15:22:32.290116+0000","last_became_peered":"2024-04-15T15:22:32.290116+0000","last_unstale":"2024-04-15T15:22:32.293378+0000","last_undegraded":"2024-04-15T15:22:32.293378+0000","last_fullsized":"2024-04-15T15:22:32.293378+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:28:10.589579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.280827+0000","last_change":"2024-04-15T15:22:32.280827+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.280827+0000","last_undegraded":"2024-04-15T15:22:32.280827+0000","last_fullsized":"2024-04-15T15:22:32.280827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303060+0000","last_change":"2024-04-15T15:22:32.299094+0000","last_active":"2024-04-15T15:22:32.303060+0000","last_peered":"2024-04-15T15:22:32.303060+0000","last_clean":"2024-04-15T15:22:32.303060+0000","last_became_active":"2024-04-15T15:22:32.298792+0000","last_became_peered":"2024-04-15T15:22:32.298792+0000","last_unstale":"2024-04-15T15:22:32.303060+0000","last_undegraded":"2024-04-15T15:22:32.303060+0000","last_fullsized":"2024-04-15T15:22:32.303060+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:46:18.965814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284940+0000","last_change":"2024-04-15T15:22:32.284940+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284940+0000","last_undegraded":"2024-04-15T15:22:32.284940+0000","last_fullsized":"2024-04-15T15:22:32.284940+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301061+0000","last_change":"2024-04-15T15:22:32.298790+0000","last_active":"2024-04-15T15:22:32.301061+0000","last_peered":"2024-04-15T15:22:32.301061+0000","last_clean":"2024-04-15T15:22:32.301061+0000","last_became_active":"2024-04-15T15:22:32.298475+0000","last_became_peered":"2024-04-15T15:22:32.298475+0000","last_unstale":"2024-04-15T15:22:32.301061+0000","last_undegraded":"2024-04-15T15:22:32.301061+0000","last_fullsized":"2024-04-15T15:22:32.301061+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281646+0000","last_change":"2024-04-15T15:22:32.281646+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281646+0000","last_undegraded":"2024-04-15T15:22:32.281646+0000","last_fullsized":"2024-04-15T15:22:32.281646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.294638+0000","last_change":"2024-04-15T15:22:32.293012+0000","last_active":"2024-04-15T15:22:32.294638+0000","last_peered":"2024-04-15T15:22:32.294638+0000","last_clean":"2024-04-15T15:22:32.294638+0000","last_became_active":"2024-04-15T15:22:32.292728+0000","last_became_peered":"2024-04-15T15:22:32.292728+0000","last_unstale":"2024-04-15T15:22:32.294638+0000","last_undegraded":"2024-04-15T15:22:32.294638+0000","last_fullsized":"2024-04-15T15:22:32.294638+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:44.182373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285229+0000","last_change":"2024-04-15T15:22:32.285229+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285229+0000","last_undegraded":"2024-04-15T15:22:32.285229+0000","last_fullsized":"2024-04-15T15:22:32.285229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301126+0000","last_change":"2024-04-15T15:22:32.295146+0000","last_active":"2024-04-15T15:22:32.301126+0000","last_peered":"2024-04-15T15:22:32.301126+0000","last_clean":"2024-04-15T15:22:32.301126+0000","last_became_active":"2024-04-15T15:22:32.294842+0000","last_became_peered":"2024-04-15T15:22:32.294842+0000","last_unstale":"2024-04-15T15:22:32.301126+0000","last_undegraded":"2024-04-15T15:22:32.301126+0000","last_fullsized":"2024-04-15T15:22:32.301126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288112+0000","last_change":"2024-04-15T15:22:32.288112+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288112+0000","last_undegraded":"2024-04-15T15:22:32.288112+0000","last_fullsized":"2024-04-15T15:22:32.288112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.296985+0000","last_change":"2024-04-15T15:22:32.295490+0000","last_active":"2024-04-15T15:22:32.296985+0000","last_peered":"2024-04-15T15:22:32.296985+0000","last_clean":"2024-04-15T15:22:32.296985+0000","last_became_active":"2024-04-15T15:22:32.295129+0000","last_became_peered":"2024-04-15T15:22:32.295129+0000","last_unstale":"2024-04-15T15:22:32.296985+0000","last_undegraded":"2024-04-15T15:22:32.296985+0000","last_fullsized":"2024-04-15T15:22:32.296985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:26:37.527479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282394+0000","last_change":"2024-04-15T15:22:32.282394+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282394+0000","last_undegraded":"2024-04-15T15:22:32.282394+0000","last_fullsized":"2024-04-15T15:22:32.282394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.295606+0000","last_change":"2024-04-15T15:22:32.294246+0000","last_active":"2024-04-15T15:22:32.295606+0000","last_peered":"2024-04-15T15:22:32.295606+0000","last_clean":"2024-04-15T15:22:32.295606+0000","last_became_active":"2024-04-15T15:22:32.293952+0000","last_became_peered":"2024-04-15T15:22:32.293952+0000","last_unstale":"2024-04-15T15:22:32.295606+0000","last_undegraded":"2024-04-15T15:22:32.295606+0000","last_fullsized":"2024-04-15T15:22:32.295606+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.280827+0000","last_change":"2024-04-15T15:22:32.280827+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.280827+0000","last_undegraded":"2024-04-15T15:22:32.280827+0000","last_fullsized":"2024-04-15T15:22:32.280827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.292565+0000","last_change":"2024-04-15T15:22:32.288288+0000","last_active":"2024-04-15T15:22:32.292565+0000","last_peered":"2024-04-15T15:22:32.292565+0000","last_clean":"2024-04-15T15:22:32.292565+0000","last_became_active":"2024-04-15T15:22:32.287969+0000","last_became_peered":"2024-04-15T15:22:32.287969+0000","last_unstale":"2024-04-15T15:22:32.292565+0000","last_undegraded":"2024-04-15T15:22:32.292565+0000","last_fullsized":"2024-04-15T15:22:32.292565+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286818+0000","last_change":"2024-04-15T15:22:32.286818+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286818+0000","last_undegraded":"2024-04-15T15:22:32.286818+0000","last_fullsized":"2024-04-15T15:22:32.286818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.294372+0000","last_change":"2024-04-15T15:22:32.292570+0000","last_active":"2024-04-15T15:22:32.294372+0000","last_peered":"2024-04-15T15:22:32.294372+0000","last_clean":"2024-04-15T15:22:32.294372+0000","last_became_active":"2024-04-15T15:22:32.292275+0000","last_became_peered":"2024-04-15T15:22:32.292275+0000","last_unstale":"2024-04-15T15:22:32.294372+0000","last_undegraded":"2024-04-15T15:22:32.294372+0000","last_fullsized":"2024-04-15T15:22:32.294372+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:10:15.989931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283018+0000","last_change":"2024-04-15T15:22:32.283018+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283018+0000","last_undegraded":"2024-04-15T15:22:32.283018+0000","last_fullsized":"2024-04-15T15:22:32.283018+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301534+0000","last_change":"2024-04-15T15:22:32.297615+0000","last_active":"2024-04-15T15:22:32.301534+0000","last_peered":"2024-04-15T15:22:32.301534+0000","last_clean":"2024-04-15T15:22:32.301534+0000","last_became_active":"2024-04-15T15:22:32.297385+0000","last_became_peered":"2024-04-15T15:22:32.297385+0000","last_unstale":"2024-04-15T15:22:32.301534+0000","last_undegraded":"2024-04-15T15:22:32.301534+0000","last_fullsized":"2024-04-15T15:22:32.301534+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:05:16.011118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287505+0000","last_change":"2024-04-15T15:22:32.287505+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287505+0000","last_undegraded":"2024-04-15T15:22:32.287505+0000","last_fullsized":"2024-04-15T15:22:32.287505+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293952+0000","last_change":"2024-04-15T15:22:32.292214+0000","last_active":"2024-04-15T15:22:32.293952+0000","last_peered":"2024-04-15T15:22:32.293952+0000","last_clean":"2024-04-15T15:22:32.293952+0000","last_became_active":"2024-04-15T15:22:32.291892+0000","last_became_peered":"2024-04-15T15:22:32.291892+0000","last_unstale":"2024-04-15T15:22:32.293952+0000","last_undegraded":"2024-04-15T15:22:32.293952+0000","last_fullsized":"2024-04-15T15:22:32.293952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285960+0000","last_change":"2024-04-15T15:22:32.285960+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285960+0000","last_undegraded":"2024-04-15T15:22:32.285960+0000","last_fullsized":"2024-04-15T15:22:32.285960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300939+0000","last_change":"2024-04-15T15:22:32.299060+0000","last_active":"2024-04-15T15:22:32.300939+0000","last_peered":"2024-04-15T15:22:32.300939+0000","last_clean":"2024-04-15T15:22:32.300939+0000","last_became_active":"2024-04-15T15:22:32.298599+0000","last_became_peered":"2024-04-15T15:22:32.298599+0000","last_unstale":"2024-04-15T15:22:32.300939+0000","last_undegraded":"2024-04-15T15:22:32.300939+0000","last_fullsized":"2024-04-15T15:22:32.300939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.289893+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:22:32.289893+0000","last_peered":"2024-04-15T15:22:32.289893+0000","last_clean":"2024-04-15T15:22:32.289893+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:22:32.289893+0000","last_undegraded":"2024-04-15T15:22:32.289893+0000","last_fullsized":"2024-04-15T15:22:32.289893+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":771540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":764873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:22:37.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:22:37.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.819+0000 7fcc3eb15640 1 Processor -- start 2024-04-15T15:22:37.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.819+0000 7fcc3eb15640 1 -- start start 2024-04-15T15:22:37.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.820+0000 7fcc3eb15640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcc3809b880 0x7fcc38070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.820+0000 7fcc3eb15640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 0x7fcc380d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.820+0000 7fcc3eb15640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc380d05e0 con 0x7fcc38070940 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.820+0000 7fcc3eb15640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fcc3809bce0 con 0x7fcc3809b880 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.820+0000 7fcc3eb15640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcc38071260 con 0x7fcc380d0da0 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.820+0000 7fcc3c8b1640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcc38070940 0x7fcc38160310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35948/0 (socket says 172.21.15.12:35948) 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.820+0000 7fcc3c8b1640 1 -- 172.21.15.12:0/393356313 learned_addr learned my addr 172.21.15.12:0/393356313 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc37fff640 1 --2- 172.21.15.12:0/393356313 >> v2:172.21.15.55:3300/0 conn(0x7fcc3809b880 0x7fcc38070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3d0b2640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 0x7fcc380d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 425783598 0 0) 0x7fcc380d05e0 con 0x7fcc38070940 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc20003520 con 0x7fcc38070940 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 588658470 0 0) 0x7fcc20003520 con 0x7fcc38070940 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 msgr2=0x7fcc380d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 0x7fcc380d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 >> v2:172.21.15.55:3300/0 conn(0x7fcc3809b880 msgr2=0x7fcc38070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 --2- 172.21.15.12:0/393356313 >> v2:172.21.15.55:3300/0 conn(0x7fcc3809b880 0x7fcc38070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3815f720 con 0x7fcc38070940 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fcc28001670 con 0x7fcc38070940 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc37fff640 1 --2- 172.21.15.12:0/393356313 >> v2:172.21.15.55:3300/0 conn(0x7fcc3809b880 0x7fcc38070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 >> v1:172.21.15.12:6789/0 conn(0x7fcc38070940 legacy=0x7fcc38160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc20004800 0x7fcc20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 0x7fcc200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3809bce0 con 0x7fcc200039b0 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcc38071260 con 0x7fcc20004800 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcc2000b9c0 con 0x7fcc380d0da0 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3c8b1640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc37fff640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc20004800 0x7fcc20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3d0b2640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 0x7fcc200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3c8b1640 1 -- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 msgr2=0x7fcc200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3c8b1640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 0x7fcc200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3c8b1640 1 -- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc20004800 msgr2=0x7fcc20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3c8b1640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc20004800 0x7fcc20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3c8b1640 1 -- 172.21.15.12:0/393356313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc20003520 con 0x7fcc200039b0 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.821+0000 7fcc3d0b2640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380d0da0 0x7fcc200051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.822+0000 7fcc3c8b1640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc20003dd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcc28001050 tx=0x7fcc20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c126b5e0a50d0c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.822+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28007d70 con 0x7fcc200039b0 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.822+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc28030070 con 0x7fcc200039b0 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.822+0000 7fcc377fe640 1 -- 172.21.15.12:0/393356313 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2802cd80 con 0x7fcc200039b0 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.823+0000 7fcc3eb15640 1 -- 172.21.15.12:0/393356313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 msgr2=0x7fcc20003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.823+0000 7fcc3eb15640 1 --2- 172.21.15.12:0/393356313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc20003dd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcc28001050 tx=0x7fcc20005830 comp rx=0 tx=0).stop 2024-04-15T15:22:37.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.823+0000 7fcc3d0b2640 1 -- 172.21.15.12:0/393356313 reap_dead start 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.823+0000 7fcc3eb15640 1 -- 172.21.15.12:0/393356313 shutdown_connections 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.823+0000 7fcc3eb15640 1 -- 172.21.15.12:0/393356313 >> 172.21.15.12:0/393356313 conn(0x7fcc380b3f40 msgr2=0x7fcc380b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.823+0000 7fcc3eb15640 1 -- 172.21.15.12:0/393356313 shutdown_connections 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.823+0000 7fcc3eb15640 1 -- 172.21.15.12:0/393356313 wait complete. 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.824+0000 7fcc3eb15640 1 Processor -- start 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.824+0000 7fcc3eb15640 1 -- start start 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3eb15640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc38152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3eb15640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380b7b40 0x7fcc38157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3eb15640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc38158490 0x7fcc38158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3eb15640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcc380d0880 con 0x7fcc200039b0 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3eb15640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3815fce0 con 0x7fcc38158490 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3eb15640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcc380d1210 con 0x7fcc380b7b40 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3c8b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc38152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3c8b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc38152430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49536/0 (socket says 172.21.15.12:49536) 2024-04-15T15:22:37.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3c8b1640 1 -- 172.21.15.12:0/767142866 learned_addr learned my addr 172.21.15.12:0/767142866 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc37fff640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380b7b40 0x7fcc38157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3d0b2640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc38158490 0x7fcc38158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3c8b1640 1 -- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380b7b40 msgr2=0x7fcc38157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3c8b1640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380b7b40 0x7fcc38157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3c8b1640 1 -- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc38158490 msgr2=0x7fcc38158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3c8b1640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc38158490 0x7fcc38158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3c8b1640 1 -- 172.21.15.12:0/767142866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc38152e30 con 0x7fcc200039b0 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc3d0b2640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc38158490 0x7fcc38158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.825+0000 7fcc37fff640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380b7b40 0x7fcc38157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.826+0000 7fcc3c8b1640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc38152430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800c2d0 tx=0x7fcc28000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.826+0000 7fcc357fa640 1 -- 172.21.15.12:0/767142866 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28004bf0 con 0x7fcc200039b0 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.826+0000 7fcc357fa640 1 -- 172.21.15.12:0/767142866 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc28004d90 con 0x7fcc200039b0 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.826+0000 7fcc357fa640 1 -- 172.21.15.12:0/767142866 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28007270 con 0x7fcc200039b0 2024-04-15T15:22:37.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.826+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc38158f20 con 0x7fcc200039b0 2024-04-15T15:22:37.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.826+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc380b5340 con 0x7fcc200039b0 2024-04-15T15:22:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.827+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc380689e0 con 0x7fcc200039b0 2024-04-15T15:22:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.827+0000 7fcc357fa640 1 -- 172.21.15.12:0/767142866 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcc28047020 con 0x7fcc200039b0 2024-04-15T15:22:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.828+0000 7fcc357fa640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcc2406c920 0x7fcc2406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.828+0000 7fcc357fa640 1 -- 172.21.15.12:0/767142866 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcc280acf20 con 0x7fcc200039b0 2024-04-15T15:22:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.828+0000 7fcc37fff640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcc2406c920 0x7fcc2406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.829+0000 7fcc37fff640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcc2406c920 0x7fcc2406ede0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c006d40 tx=0x7fcc2c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:37.832+0000 7fcc357fa640 1 -- 172.21.15.12:0/767142866 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcc2807b1c0 con 0x7fcc200039b0 2024-04-15T15:22:38.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.013+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc380cea30 con 0x7fcc2406c920 2024-04-15T15:22:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.019+0000 7fcc357fa640 1 -- 172.21.15.12:0/767142866 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7fcc380cea30 con 0x7fcc2406c920 2024-04-15T15:22:38.024 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:22:38.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcc2406c920 msgr2=0x7fcc2406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcc2406c920 0x7fcc2406ede0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c006d40 tx=0x7fcc2c008040 comp rx=0 tx=0).stop 2024-04-15T15:22:38.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 msgr2=0x7fcc38152430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc38152430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800c2d0 tx=0x7fcc28000fe0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 shutdown_connections 2024-04-15T15:22:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcc2406c920 0x7fcc2406ede0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc38158490 0x7fcc38158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcc380b7b40 0x7fcc38157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 --2- 172.21.15.12:0/767142866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc200039b0 0x7fcc38152430 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.022+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 >> 172.21.15.12:0/767142866 conn(0x7fcc380b3f40 msgr2=0x7fcc3815e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.023+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 shutdown_connections 2024-04-15T15:22:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.023+0000 7fcc3eb15640 1 -- 172.21.15.12:0/767142866 wait complete. 2024-04-15T15:22:38.036 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-15T15:22:36.608251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237216,"kb_used_data":4416,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737504,"statfs":{"total":773094113280,"available":772851204096,"internally_reserved":0,"allocated":4521984,"data_stored":2590110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.331612"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283803+0000","last_change":"2024-04-15T15:22:32.283803+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283803+0000","last_undegraded":"2024-04-15T15:22:32.283803+0000","last_fullsized":"2024-04-15T15:22:32.283803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282846+0000","last_change":"2024-04-15T15:22:32.282846+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282846+0000","last_undegraded":"2024-04-15T15:22:32.282846+0000","last_fullsized":"2024-04-15T15:22:32.282846+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282557+0000","last_change":"2024-04-15T15:22:32.282557+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282557+0000","last_undegraded":"2024-04-15T15:22:32.282557+0000","last_fullsized":"2024-04-15T15:22:32.282557+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285184+0000","last_change":"2024-04-15T15:22:32.285184+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285184+0000","last_undegraded":"2024-04-15T15:22:32.285184+0000","last_fullsized":"2024-04-15T15:22:32.285184+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285654+0000","last_change":"2024-04-15T15:22:32.285654+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285654+0000","last_undegraded":"2024-04-15T15:22:32.285654+0000","last_fullsized":"2024-04-15T15:22:32.285654+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288873+0000","last_change":"2024-04-15T15:22:32.288873+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288873+0000","last_undegraded":"2024-04-15T15:22:32.288873+0000","last_fullsized":"2024-04-15T15:22:32.288873+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.289028+0000","last_change":"2024-04-15T15:22:32.289028+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.289028+0000","last_undegraded":"2024-04-15T15:22:32.289028+0000","last_fullsized":"2024-04-15T15:22:32.289028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281732+0000","last_change":"2024-04-15T15:22:32.281732+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281732+0000","last_undegraded":"2024-04-15T15:22:32.281732+0000","last_fullsized":"2024-04-15T15:22:32.281732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282504+0000","last_change":"2024-04-15T15:22:32.282504+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282504+0000","last_undegraded":"2024-04-15T15:22:32.282504+0000","last_fullsized":"2024-04-15T15:22:32.282504+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284678+0000","last_change":"2024-04-15T15:22:32.284678+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284678+0000","last_undegraded":"2024-04-15T15:22:32.284678+0000","last_fullsized":"2024-04-15T15:22:32.284678+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286253+0000","last_change":"2024-04-15T15:22:32.286253+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286253+0000","last_undegraded":"2024-04-15T15:22:32.286253+0000","last_fullsized":"2024-04-15T15:22:32.286253+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282565+0000","last_change":"2024-04-15T15:22:32.282565+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282565+0000","last_undegraded":"2024-04-15T15:22:32.282565+0000","last_fullsized":"2024-04-15T15:22:32.282565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.289197+0000","last_change":"2024-04-15T15:22:32.289197+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.289197+0000","last_undegraded":"2024-04-15T15:22:32.289197+0000","last_fullsized":"2024-04-15T15:22:32.289197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286855+0000","last_change":"2024-04-15T15:22:32.286855+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286855+0000","last_undegraded":"2024-04-15T15:22:32.286855+0000","last_fullsized":"2024-04-15T15:22:32.286855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288550+0000","last_change":"2024-04-15T15:22:32.288550+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288550+0000","last_undegraded":"2024-04-15T15:22:32.288550+0000","last_fullsized":"2024-04-15T15:22:32.288550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287794+0000","last_change":"2024-04-15T15:22:32.287794+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287794+0000","last_undegraded":"2024-04-15T15:22:32.287794+0000","last_fullsized":"2024-04-15T15:22:32.287794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287169+0000","last_change":"2024-04-15T15:22:32.287169+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287169+0000","last_undegraded":"2024-04-15T15:22:32.287169+0000","last_fullsized":"2024-04-15T15:22:32.287169+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283134+0000","last_change":"2024-04-15T15:22:32.283134+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283134+0000","last_undegraded":"2024-04-15T15:22:32.283134+0000","last_fullsized":"2024-04-15T15:22:32.283134+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287033+0000","last_change":"2024-04-15T15:22:32.287033+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287033+0000","last_undegraded":"2024-04-15T15:22:32.287033+0000","last_fullsized":"2024-04-15T15:22:32.287033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283165+0000","last_change":"2024-04-15T15:22:32.283165+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283165+0000","last_undegraded":"2024-04-15T15:22:32.283165+0000","last_fullsized":"2024-04-15T15:22:32.283165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287661+0000","last_change":"2024-04-15T15:22:32.287661+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287661+0000","last_undegraded":"2024-04-15T15:22:32.287661+0000","last_fullsized":"2024-04-15T15:22:32.287661+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282573+0000","last_change":"2024-04-15T15:22:32.282573+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282573+0000","last_undegraded":"2024-04-15T15:22:32.282573+0000","last_fullsized":"2024-04-15T15:22:32.282573+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288692+0000","last_change":"2024-04-15T15:22:32.288692+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288692+0000","last_undegraded":"2024-04-15T15:22:32.288692+0000","last_fullsized":"2024-04-15T15:22:32.288692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286380+0000","last_change":"2024-04-15T15:22:32.286380+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286380+0000","last_undegraded":"2024-04-15T15:22:32.286380+0000","last_fullsized":"2024-04-15T15:22:32.286380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283560+0000","last_change":"2024-04-15T15:22:32.283560+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283560+0000","last_undegraded":"2024-04-15T15:22:32.283560+0000","last_fullsized":"2024-04-15T15:22:32.283560+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286540+0000","last_change":"2024-04-15T15:22:32.286540+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286540+0000","last_undegraded":"2024-04-15T15:22:32.286540+0000","last_fullsized":"2024-04-15T15:22:32.286540+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282295+0000","last_change":"2024-04-15T15:22:32.282295+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282295+0000","last_undegraded":"2024-04-15T15:22:32.282295+0000","last_fullsized":"2024-04-15T15:22:32.282295+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287436+0000","last_change":"2024-04-15T15:22:32.287436+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287436+0000","last_undegraded":"2024-04-15T15:22:32.287436+0000","last_fullsized":"2024-04-15T15:22:32.287436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284239+0000","last_change":"2024-04-15T15:22:32.284239+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284239+0000","last_undegraded":"2024-04-15T15:22:32.284239+0000","last_fullsized":"2024-04-15T15:22:32.284239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283378+0000","last_change":"2024-04-15T15:22:32.283378+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283378+0000","last_undegraded":"2024-04-15T15:22:32.283378+0000","last_fullsized":"2024-04-15T15:22:32.283378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286533+0000","last_change":"2024-04-15T15:22:32.286533+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286533+0000","last_undegraded":"2024-04-15T15:22:32.286533+0000","last_fullsized":"2024-04-15T15:22:32.286533+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286417+0000","last_change":"2024-04-15T15:22:32.286417+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286417+0000","last_undegraded":"2024-04-15T15:22:32.286417+0000","last_fullsized":"2024-04-15T15:22:32.286417+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287816+0000","last_change":"2024-04-15T15:22:32.287816+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287816+0000","last_undegraded":"2024-04-15T15:22:32.287816+0000","last_fullsized":"2024-04-15T15:22:32.287816+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300082+0000","last_change":"2024-04-15T15:22:32.295369+0000","last_active":"2024-04-15T15:22:32.300082+0000","last_peered":"2024-04-15T15:22:32.300082+0000","last_clean":"2024-04-15T15:22:32.300082+0000","last_became_active":"2024-04-15T15:22:32.295054+0000","last_became_peered":"2024-04-15T15:22:32.295054+0000","last_unstale":"2024-04-15T15:22:32.300082+0000","last_undegraded":"2024-04-15T15:22:32.300082+0000","last_fullsized":"2024-04-15T15:22:32.300082+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:05:16.011118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288081+0000","last_change":"2024-04-15T15:22:32.288081+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288081+0000","last_undegraded":"2024-04-15T15:22:32.288081+0000","last_fullsized":"2024-04-15T15:22:32.288081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303013+0000","last_change":"2024-04-15T15:22:32.298663+0000","last_active":"2024-04-15T15:22:32.303013+0000","last_peered":"2024-04-15T15:22:32.303013+0000","last_clean":"2024-04-15T15:22:32.303013+0000","last_became_active":"2024-04-15T15:22:32.298296+0000","last_became_peered":"2024-04-15T15:22:32.298296+0000","last_unstale":"2024-04-15T15:22:32.303013+0000","last_undegraded":"2024-04-15T15:22:32.303013+0000","last_fullsized":"2024-04-15T15:22:32.303013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284349+0000","last_change":"2024-04-15T15:22:32.284349+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284349+0000","last_undegraded":"2024-04-15T15:22:32.284349+0000","last_fullsized":"2024-04-15T15:22:32.284349+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306108+0000","last_change":"2024-04-15T15:22:32.301621+0000","last_active":"2024-04-15T15:22:32.306108+0000","last_peered":"2024-04-15T15:22:32.306108+0000","last_clean":"2024-04-15T15:22:32.306108+0000","last_became_active":"2024-04-15T15:22:32.301253+0000","last_became_peered":"2024-04-15T15:22:32.301253+0000","last_unstale":"2024-04-15T15:22:32.306108+0000","last_undegraded":"2024-04-15T15:22:32.306108+0000","last_fullsized":"2024-04-15T15:22:32.306108+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283316+0000","last_change":"2024-04-15T15:22:32.283316+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283316+0000","last_undegraded":"2024-04-15T15:22:32.283316+0000","last_fullsized":"2024-04-15T15:22:32.283316+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301630+0000","last_change":"2024-04-15T15:22:32.297657+0000","last_active":"2024-04-15T15:22:32.301630+0000","last_peered":"2024-04-15T15:22:32.301630+0000","last_clean":"2024-04-15T15:22:32.301630+0000","last_became_active":"2024-04-15T15:22:32.297419+0000","last_became_peered":"2024-04-15T15:22:32.297419+0000","last_unstale":"2024-04-15T15:22:32.301630+0000","last_undegraded":"2024-04-15T15:22:32.301630+0000","last_fullsized":"2024-04-15T15:22:32.301630+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:09:33.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282144+0000","last_change":"2024-04-15T15:22:32.282144+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282144+0000","last_undegraded":"2024-04-15T15:22:32.282144+0000","last_fullsized":"2024-04-15T15:22:32.282144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300355+0000","last_change":"2024-04-15T15:22:32.293808+0000","last_active":"2024-04-15T15:22:32.300355+0000","last_peered":"2024-04-15T15:22:32.300355+0000","last_clean":"2024-04-15T15:22:32.300355+0000","last_became_active":"2024-04-15T15:22:32.293474+0000","last_became_peered":"2024-04-15T15:22:32.293474+0000","last_unstale":"2024-04-15T15:22:32.300355+0000","last_undegraded":"2024-04-15T15:22:32.300355+0000","last_fullsized":"2024-04-15T15:22:32.300355+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287768+0000","last_change":"2024-04-15T15:22:32.287768+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287768+0000","last_undegraded":"2024-04-15T15:22:32.287768+0000","last_fullsized":"2024-04-15T15:22:32.287768+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301034+0000","last_change":"2024-04-15T15:22:32.299916+0000","last_active":"2024-04-15T15:22:32.301034+0000","last_peered":"2024-04-15T15:22:32.301034+0000","last_clean":"2024-04-15T15:22:32.301034+0000","last_became_active":"2024-04-15T15:22:32.299576+0000","last_became_peered":"2024-04-15T15:22:32.299576+0000","last_unstale":"2024-04-15T15:22:32.301034+0000","last_undegraded":"2024-04-15T15:22:32.301034+0000","last_fullsized":"2024-04-15T15:22:32.301034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282000+0000","last_change":"2024-04-15T15:22:32.282000+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282000+0000","last_undegraded":"2024-04-15T15:22:32.282000+0000","last_fullsized":"2024-04-15T15:22:32.282000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306074+0000","last_change":"2024-04-15T15:22:32.301232+0000","last_active":"2024-04-15T15:22:32.306074+0000","last_peered":"2024-04-15T15:22:32.306074+0000","last_clean":"2024-04-15T15:22:32.306074+0000","last_became_active":"2024-04-15T15:22:32.300885+0000","last_became_peered":"2024-04-15T15:22:32.300885+0000","last_unstale":"2024-04-15T15:22:32.306074+0000","last_undegraded":"2024-04-15T15:22:32.306074+0000","last_fullsized":"2024-04-15T15:22:32.306074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:01.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282531+0000","last_change":"2024-04-15T15:22:32.282531+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282531+0000","last_undegraded":"2024-04-15T15:22:32.282531+0000","last_fullsized":"2024-04-15T15:22:32.282531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293532+0000","last_change":"2024-04-15T15:22:32.288888+0000","last_active":"2024-04-15T15:22:32.293532+0000","last_peered":"2024-04-15T15:22:32.293532+0000","last_clean":"2024-04-15T15:22:32.293532+0000","last_became_active":"2024-04-15T15:22:32.288559+0000","last_became_peered":"2024-04-15T15:22:32.288559+0000","last_unstale":"2024-04-15T15:22:32.293532+0000","last_undegraded":"2024-04-15T15:22:32.293532+0000","last_fullsized":"2024-04-15T15:22:32.293532+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:02:52.088900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284589+0000","last_change":"2024-04-15T15:22:32.284589+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284589+0000","last_undegraded":"2024-04-15T15:22:32.284589+0000","last_fullsized":"2024-04-15T15:22:32.284589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293989+0000","last_change":"2024-04-15T15:22:32.292353+0000","last_active":"2024-04-15T15:22:32.293989+0000","last_peered":"2024-04-15T15:22:32.293989+0000","last_clean":"2024-04-15T15:22:32.293989+0000","last_became_active":"2024-04-15T15:22:32.292043+0000","last_became_peered":"2024-04-15T15:22:32.292043+0000","last_unstale":"2024-04-15T15:22:32.293989+0000","last_undegraded":"2024-04-15T15:22:32.293989+0000","last_fullsized":"2024-04-15T15:22:32.293989+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:21:12.998104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288577+0000","last_change":"2024-04-15T15:22:32.288577+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288577+0000","last_undegraded":"2024-04-15T15:22:32.288577+0000","last_fullsized":"2024-04-15T15:22:32.288577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301133+0000","last_change":"2024-04-15T15:22:32.294706+0000","last_active":"2024-04-15T15:22:32.301133+0000","last_peered":"2024-04-15T15:22:32.301133+0000","last_clean":"2024-04-15T15:22:32.301133+0000","last_became_active":"2024-04-15T15:22:32.294353+0000","last_became_peered":"2024-04-15T15:22:32.294353+0000","last_unstale":"2024-04-15T15:22:32.301133+0000","last_undegraded":"2024-04-15T15:22:32.301133+0000","last_fullsized":"2024-04-15T15:22:32.301133+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281997+0000","last_change":"2024-04-15T15:22:32.281997+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281997+0000","last_undegraded":"2024-04-15T15:22:32.281997+0000","last_fullsized":"2024-04-15T15:22:32.281997+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301078+0000","last_change":"2024-04-15T15:22:32.294321+0000","last_active":"2024-04-15T15:22:32.301078+0000","last_peered":"2024-04-15T15:22:32.301078+0000","last_clean":"2024-04-15T15:22:32.301078+0000","last_became_active":"2024-04-15T15:22:32.294043+0000","last_became_peered":"2024-04-15T15:22:32.294043+0000","last_unstale":"2024-04-15T15:22:32.301078+0000","last_undegraded":"2024-04-15T15:22:32.301078+0000","last_fullsized":"2024-04-15T15:22:32.301078+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284953+0000","last_change":"2024-04-15T15:22:32.284953+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284953+0000","last_undegraded":"2024-04-15T15:22:32.284953+0000","last_fullsized":"2024-04-15T15:22:32.284953+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.305720+0000","last_change":"2024-04-15T15:22:32.298894+0000","last_active":"2024-04-15T15:22:32.305720+0000","last_peered":"2024-04-15T15:22:32.305720+0000","last_clean":"2024-04-15T15:22:32.305720+0000","last_became_active":"2024-04-15T15:22:32.298534+0000","last_became_peered":"2024-04-15T15:22:32.298534+0000","last_unstale":"2024-04-15T15:22:32.305720+0000","last_undegraded":"2024-04-15T15:22:32.305720+0000","last_fullsized":"2024-04-15T15:22:32.305720+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:45.706167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281544+0000","last_change":"2024-04-15T15:22:32.281544+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281544+0000","last_undegraded":"2024-04-15T15:22:32.281544+0000","last_fullsized":"2024-04-15T15:22:32.281544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.292179+0000","last_change":"2024-04-15T15:22:32.287177+0000","last_active":"2024-04-15T15:22:32.292179+0000","last_peered":"2024-04-15T15:22:32.292179+0000","last_clean":"2024-04-15T15:22:32.292179+0000","last_became_active":"2024-04-15T15:22:32.286887+0000","last_became_peered":"2024-04-15T15:22:32.286887+0000","last_unstale":"2024-04-15T15:22:32.292179+0000","last_undegraded":"2024-04-15T15:22:32.292179+0000","last_fullsized":"2024-04-15T15:22:32.292179+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281381+0000","last_change":"2024-04-15T15:22:32.281381+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281381+0000","last_undegraded":"2024-04-15T15:22:32.281381+0000","last_fullsized":"2024-04-15T15:22:32.281381+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306066+0000","last_change":"2024-04-15T15:22:32.301053+0000","last_active":"2024-04-15T15:22:32.306066+0000","last_peered":"2024-04-15T15:22:32.306066+0000","last_clean":"2024-04-15T15:22:32.306066+0000","last_became_active":"2024-04-15T15:22:32.300755+0000","last_became_peered":"2024-04-15T15:22:32.300755+0000","last_unstale":"2024-04-15T15:22:32.306066+0000","last_undegraded":"2024-04-15T15:22:32.306066+0000","last_fullsized":"2024-04-15T15:22:32.306066+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:09:33.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285849+0000","last_change":"2024-04-15T15:22:32.285849+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285849+0000","last_undegraded":"2024-04-15T15:22:32.285849+0000","last_fullsized":"2024-04-15T15:22:32.285849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.295745+0000","last_change":"2024-04-15T15:22:32.293921+0000","last_active":"2024-04-15T15:22:32.295745+0000","last_peered":"2024-04-15T15:22:32.295745+0000","last_clean":"2024-04-15T15:22:32.295745+0000","last_became_active":"2024-04-15T15:22:32.293641+0000","last_became_peered":"2024-04-15T15:22:32.293641+0000","last_unstale":"2024-04-15T15:22:32.295745+0000","last_undegraded":"2024-04-15T15:22:32.295745+0000","last_fullsized":"2024-04-15T15:22:32.295745+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:02:52.088900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282090+0000","last_change":"2024-04-15T15:22:32.282090+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282090+0000","last_undegraded":"2024-04-15T15:22:32.282090+0000","last_fullsized":"2024-04-15T15:22:32.282090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301077+0000","last_change":"2024-04-15T15:22:32.299339+0000","last_active":"2024-04-15T15:22:32.301077+0000","last_peered":"2024-04-15T15:22:32.301077+0000","last_clean":"2024-04-15T15:22:32.301077+0000","last_became_active":"2024-04-15T15:22:32.298921+0000","last_became_peered":"2024-04-15T15:22:32.298921+0000","last_unstale":"2024-04-15T15:22:32.301077+0000","last_undegraded":"2024-04-15T15:22:32.301077+0000","last_fullsized":"2024-04-15T15:22:32.301077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281983+0000","last_change":"2024-04-15T15:22:32.281983+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281983+0000","last_undegraded":"2024-04-15T15:22:32.281983+0000","last_fullsized":"2024-04-15T15:22:32.281983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301166+0000","last_change":"2024-04-15T15:22:32.299157+0000","last_active":"2024-04-15T15:22:32.301166+0000","last_peered":"2024-04-15T15:22:32.301166+0000","last_clean":"2024-04-15T15:22:32.301166+0000","last_became_active":"2024-04-15T15:22:32.298836+0000","last_became_peered":"2024-04-15T15:22:32.298836+0000","last_unstale":"2024-04-15T15:22:32.301166+0000","last_undegraded":"2024-04-15T15:22:32.301166+0000","last_fullsized":"2024-04-15T15:22:32.301166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285787+0000","last_change":"2024-04-15T15:22:32.285787+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285787+0000","last_undegraded":"2024-04-15T15:22:32.285787+0000","last_fullsized":"2024-04-15T15:22:32.285787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.302895+0000","last_change":"2024-04-15T15:22:32.301976+0000","last_active":"2024-04-15T15:22:32.302895+0000","last_peered":"2024-04-15T15:22:32.302895+0000","last_clean":"2024-04-15T15:22:32.302895+0000","last_became_active":"2024-04-15T15:22:32.301692+0000","last_became_peered":"2024-04-15T15:22:32.301692+0000","last_unstale":"2024-04-15T15:22:32.302895+0000","last_undegraded":"2024-04-15T15:22:32.302895+0000","last_fullsized":"2024-04-15T15:22:32.302895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:01.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284873+0000","last_change":"2024-04-15T15:22:32.284873+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284873+0000","last_undegraded":"2024-04-15T15:22:32.284873+0000","last_fullsized":"2024-04-15T15:22:32.284873+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303068+0000","last_change":"2024-04-15T15:22:32.299415+0000","last_active":"2024-04-15T15:22:32.303068+0000","last_peered":"2024-04-15T15:22:32.303068+0000","last_clean":"2024-04-15T15:22:32.303068+0000","last_became_active":"2024-04-15T15:22:32.299078+0000","last_became_peered":"2024-04-15T15:22:32.299078+0000","last_unstale":"2024-04-15T15:22:32.303068+0000","last_undegraded":"2024-04-15T15:22:32.303068+0000","last_fullsized":"2024-04-15T15:22:32.303068+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:17:24.243206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285847+0000","last_change":"2024-04-15T15:22:32.285847+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285847+0000","last_undegraded":"2024-04-15T15:22:32.285847+0000","last_fullsized":"2024-04-15T15:22:32.285847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.306094+0000","last_change":"2024-04-15T15:22:32.301462+0000","last_active":"2024-04-15T15:22:32.306094+0000","last_peered":"2024-04-15T15:22:32.306094+0000","last_clean":"2024-04-15T15:22:32.306094+0000","last_became_active":"2024-04-15T15:22:32.301176+0000","last_became_peered":"2024-04-15T15:22:32.301176+0000","last_unstale":"2024-04-15T15:22:32.306094+0000","last_undegraded":"2024-04-15T15:22:32.306094+0000","last_fullsized":"2024-04-15T15:22:32.306094+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285710+0000","last_change":"2024-04-15T15:22:32.285710+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285710+0000","last_undegraded":"2024-04-15T15:22:32.285710+0000","last_fullsized":"2024-04-15T15:22:32.285710+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293378+0000","last_change":"2024-04-15T15:22:32.290430+0000","last_active":"2024-04-15T15:22:32.293378+0000","last_peered":"2024-04-15T15:22:32.293378+0000","last_clean":"2024-04-15T15:22:32.293378+0000","last_became_active":"2024-04-15T15:22:32.290116+0000","last_became_peered":"2024-04-15T15:22:32.290116+0000","last_unstale":"2024-04-15T15:22:32.293378+0000","last_undegraded":"2024-04-15T15:22:32.293378+0000","last_fullsized":"2024-04-15T15:22:32.293378+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:28:10.589579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.280827+0000","last_change":"2024-04-15T15:22:32.280827+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.280827+0000","last_undegraded":"2024-04-15T15:22:32.280827+0000","last_fullsized":"2024-04-15T15:22:32.280827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.303060+0000","last_change":"2024-04-15T15:22:32.299094+0000","last_active":"2024-04-15T15:22:32.303060+0000","last_peered":"2024-04-15T15:22:32.303060+0000","last_clean":"2024-04-15T15:22:32.303060+0000","last_became_active":"2024-04-15T15:22:32.298792+0000","last_became_peered":"2024-04-15T15:22:32.298792+0000","last_unstale":"2024-04-15T15:22:32.303060+0000","last_undegraded":"2024-04-15T15:22:32.303060+0000","last_fullsized":"2024-04-15T15:22:32.303060+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:46:18.965814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.284940+0000","last_change":"2024-04-15T15:22:32.284940+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.284940+0000","last_undegraded":"2024-04-15T15:22:32.284940+0000","last_fullsized":"2024-04-15T15:22:32.284940+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301061+0000","last_change":"2024-04-15T15:22:32.298790+0000","last_active":"2024-04-15T15:22:32.301061+0000","last_peered":"2024-04-15T15:22:32.301061+0000","last_clean":"2024-04-15T15:22:32.301061+0000","last_became_active":"2024-04-15T15:22:32.298475+0000","last_became_peered":"2024-04-15T15:22:32.298475+0000","last_unstale":"2024-04-15T15:22:32.301061+0000","last_undegraded":"2024-04-15T15:22:32.301061+0000","last_fullsized":"2024-04-15T15:22:32.301061+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.281646+0000","last_change":"2024-04-15T15:22:32.281646+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.281646+0000","last_undegraded":"2024-04-15T15:22:32.281646+0000","last_fullsized":"2024-04-15T15:22:32.281646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.294638+0000","last_change":"2024-04-15T15:22:32.293012+0000","last_active":"2024-04-15T15:22:32.294638+0000","last_peered":"2024-04-15T15:22:32.294638+0000","last_clean":"2024-04-15T15:22:32.294638+0000","last_became_active":"2024-04-15T15:22:32.292728+0000","last_became_peered":"2024-04-15T15:22:32.292728+0000","last_unstale":"2024-04-15T15:22:32.294638+0000","last_undegraded":"2024-04-15T15:22:32.294638+0000","last_fullsized":"2024-04-15T15:22:32.294638+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:44.182373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285229+0000","last_change":"2024-04-15T15:22:32.285229+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285229+0000","last_undegraded":"2024-04-15T15:22:32.285229+0000","last_fullsized":"2024-04-15T15:22:32.285229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301126+0000","last_change":"2024-04-15T15:22:32.295146+0000","last_active":"2024-04-15T15:22:32.301126+0000","last_peered":"2024-04-15T15:22:32.301126+0000","last_clean":"2024-04-15T15:22:32.301126+0000","last_became_active":"2024-04-15T15:22:32.294842+0000","last_became_peered":"2024-04-15T15:22:32.294842+0000","last_unstale":"2024-04-15T15:22:32.301126+0000","last_undegraded":"2024-04-15T15:22:32.301126+0000","last_fullsized":"2024-04-15T15:22:32.301126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.288112+0000","last_change":"2024-04-15T15:22:32.288112+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.288112+0000","last_undegraded":"2024-04-15T15:22:32.288112+0000","last_fullsized":"2024-04-15T15:22:32.288112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.296985+0000","last_change":"2024-04-15T15:22:32.295490+0000","last_active":"2024-04-15T15:22:32.296985+0000","last_peered":"2024-04-15T15:22:32.296985+0000","last_clean":"2024-04-15T15:22:32.296985+0000","last_became_active":"2024-04-15T15:22:32.295129+0000","last_became_peered":"2024-04-15T15:22:32.295129+0000","last_unstale":"2024-04-15T15:22:32.296985+0000","last_undegraded":"2024-04-15T15:22:32.296985+0000","last_fullsized":"2024-04-15T15:22:32.296985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:26:37.527479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.282394+0000","last_change":"2024-04-15T15:22:32.282394+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.282394+0000","last_undegraded":"2024-04-15T15:22:32.282394+0000","last_fullsized":"2024-04-15T15:22:32.282394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.295606+0000","last_change":"2024-04-15T15:22:32.294246+0000","last_active":"2024-04-15T15:22:32.295606+0000","last_peered":"2024-04-15T15:22:32.295606+0000","last_clean":"2024-04-15T15:22:32.295606+0000","last_became_active":"2024-04-15T15:22:32.293952+0000","last_became_peered":"2024-04-15T15:22:32.293952+0000","last_unstale":"2024-04-15T15:22:32.295606+0000","last_undegraded":"2024-04-15T15:22:32.295606+0000","last_fullsized":"2024-04-15T15:22:32.295606+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.280827+0000","last_change":"2024-04-15T15:22:32.280827+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.280827+0000","last_undegraded":"2024-04-15T15:22:32.280827+0000","last_fullsized":"2024-04-15T15:22:32.280827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.292565+0000","last_change":"2024-04-15T15:22:32.288288+0000","last_active":"2024-04-15T15:22:32.292565+0000","last_peered":"2024-04-15T15:22:32.292565+0000","last_clean":"2024-04-15T15:22:32.292565+0000","last_became_active":"2024-04-15T15:22:32.287969+0000","last_became_peered":"2024-04-15T15:22:32.287969+0000","last_unstale":"2024-04-15T15:22:32.292565+0000","last_undegraded":"2024-04-15T15:22:32.292565+0000","last_fullsized":"2024-04-15T15:22:32.292565+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.286818+0000","last_change":"2024-04-15T15:22:32.286818+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.286818+0000","last_undegraded":"2024-04-15T15:22:32.286818+0000","last_fullsized":"2024-04-15T15:22:32.286818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.294372+0000","last_change":"2024-04-15T15:22:32.292570+0000","last_active":"2024-04-15T15:22:32.294372+0000","last_peered":"2024-04-15T15:22:32.294372+0000","last_clean":"2024-04-15T15:22:32.294372+0000","last_became_active":"2024-04-15T15:22:32.292275+0000","last_became_peered":"2024-04-15T15:22:32.292275+0000","last_unstale":"2024-04-15T15:22:32.294372+0000","last_undegraded":"2024-04-15T15:22:32.294372+0000","last_fullsized":"2024-04-15T15:22:32.294372+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:10:15.989931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.283018+0000","last_change":"2024-04-15T15:22:32.283018+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.283018+0000","last_undegraded":"2024-04-15T15:22:32.283018+0000","last_fullsized":"2024-04-15T15:22:32.283018+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.301534+0000","last_change":"2024-04-15T15:22:32.297615+0000","last_active":"2024-04-15T15:22:32.301534+0000","last_peered":"2024-04-15T15:22:32.301534+0000","last_clean":"2024-04-15T15:22:32.301534+0000","last_became_active":"2024-04-15T15:22:32.297385+0000","last_became_peered":"2024-04-15T15:22:32.297385+0000","last_unstale":"2024-04-15T15:22:32.301534+0000","last_undegraded":"2024-04-15T15:22:32.301534+0000","last_fullsized":"2024-04-15T15:22:32.301534+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:05:16.011118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.287505+0000","last_change":"2024-04-15T15:22:32.287505+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.287505+0000","last_undegraded":"2024-04-15T15:22:32.287505+0000","last_fullsized":"2024-04-15T15:22:32.287505+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.293952+0000","last_change":"2024-04-15T15:22:32.292214+0000","last_active":"2024-04-15T15:22:32.293952+0000","last_peered":"2024-04-15T15:22:32.293952+0000","last_clean":"2024-04-15T15:22:32.293952+0000","last_became_active":"2024-04-15T15:22:32.291892+0000","last_became_peered":"2024-04-15T15:22:32.291892+0000","last_unstale":"2024-04-15T15:22:32.293952+0000","last_undegraded":"2024-04-15T15:22:32.293952+0000","last_fullsized":"2024-04-15T15:22:32.293952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:22:32.285960+0000","last_change":"2024-04-15T15:22:32.285960+0000","last_active":"2024-04-15T15:22:32.266162+0000","last_peered":"2024-04-15T15:22:32.266162+0000","last_clean":"2024-04-15T15:22:32.266162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:22:32.285960+0000","last_undegraded":"2024-04-15T15:22:32.285960+0000","last_fullsized":"2024-04-15T15:22:32.285960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.300939+0000","last_change":"2024-04-15T15:22:32.299060+0000","last_active":"2024-04-15T15:22:32.300939+0000","last_peered":"2024-04-15T15:22:32.300939+0000","last_clean":"2024-04-15T15:22:32.300939+0000","last_became_active":"2024-04-15T15:22:32.298599+0000","last_became_peered":"2024-04-15T15:22:32.298599+0000","last_unstale":"2024-04-15T15:22:32.300939+0000","last_undegraded":"2024-04-15T15:22:32.300939+0000","last_fullsized":"2024-04-15T15:22:32.300939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:22:32.289893+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:22:32.289893+0000","last_peered":"2024-04-15T15:22:32.289893+0000","last_clean":"2024-04-15T15:22:32.289893+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:22:32.289893+0000","last_undegraded":"2024-04-15T15:22:32.289893+0000","last_fullsized":"2024-04-15T15:22:32.289893+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":771540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":181172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":174505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":764873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:22:38.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.283803+0000', 'last_change': '2024-04-15T15:22:32.283803+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.283803+0000', 'last_undegraded': '2024-04-15T15:22:32.283803+0000', 'last_fullsized': '2024-04-15T15:22:32.283803+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282846+0000', 'last_change': '2024-04-15T15:22:32.282846+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282846+0000', 'last_undegraded': '2024-04-15T15:22:32.282846+0000', 'last_fullsized': '2024-04-15T15:22:32.282846+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282557+0000', 'last_change': '2024-04-15T15:22:32.282557+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282557+0000', 'last_undegraded': '2024-04-15T15:22:32.282557+0000', 'last_fullsized': '2024-04-15T15:22:32.282557+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.285184+0000', 'last_change': '2024-04-15T15:22:32.285184+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.285184+0000', 'last_undegraded': '2024-04-15T15:22:32.285184+0000', 'last_fullsized': '2024-04-15T15:22:32.285184+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.285654+0000', 'last_change': '2024-04-15T15:22:32.285654+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.285654+0000', 'last_undegraded': '2024-04-15T15:22:32.285654+0000', 'last_fullsized': '2024-04-15T15:22:32.285654+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:22:38.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.288873+0000', 'last_change': '2024-04-15T15:22:32.288873+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.288873+0000', 'last_undegraded': '2024-04-15T15:22:32.288873+0000', 'last_fullsized': '2024-04-15T15:22:32.288873+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.289028+0000', 'last_change': '2024-04-15T15:22:32.289028+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.289028+0000', 'last_undegraded': '2024-04-15T15:22:32.289028+0000', 'last_fullsized': '2024-04-15T15:22:32.289028+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.281732+0000', 'last_change': '2024-04-15T15:22:32.281732+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.281732+0000', 'last_undegraded': '2024-04-15T15:22:32.281732+0000', 'last_fullsized': '2024-04-15T15:22:32.281732+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282504+0000', 'last_change': '2024-04-15T15:22:32.282504+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282504+0000', 'last_undegraded': '2024-04-15T15:22:32.282504+0000', 'last_fullsized': '2024-04-15T15:22:32.282504+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.284678+0000', 'last_change': '2024-04-15T15:22:32.284678+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.284678+0000', 'last_undegraded': '2024-04-15T15:22:32.284678+0000', 'last_fullsized': '2024-04-15T15:22:32.284678+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2024-04-15T15:22:38.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.286253+0000', 'last_change': '2024-04-15T15:22:32.286253+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.286253+0000', 'last_undegraded': '2024-04-15T15:22:32.286253+0000', 'last_fullsized': '2024-04-15T15:22:32.286253+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282565+0000', 'last_change': '2024-04-15T15:22:32.282565+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282565+0000', 'last_undegraded': '2024-04-15T15:22:32.282565+0000', 'last_fullsized': '2024-04-15T15:22:32.282565+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.289197+0000', 'last_change': '2024-04-15T15:22:32.289197+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.289197+0000', 'last_undegraded': '2024-04-15T15:22:32.289197+0000', 'last_fullsized': '2024-04-15T15:22:32.289197+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.286855+0000', 'last_change': '2024-04-15T15:22:32.286855+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.286855+0000', 'last_undegraded': '2024-04-15T15:22:32.286855+0000', 'last_fullsized': '2024-04-15T15:22:32.286855+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.288550+0000', 'last_change': '2024-04-15T15:22:32.288550+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.288550+0000', 'last_undegraded': '2024-04-15T15:22:32.288550+0000', 'last_fullsized': '2024-04-15T15:22:32.288550+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-04-15T15:22:38.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.287794+0000', 'last_change': '2024-04-15T15:22:32.287794+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.287794+0000', 'last_undegraded': '2024-04-15T15:22:32.287794+0000', 'last_fullsized': '2024-04-15T15:22:32.287794+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-04-15T15:22:38.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.287169+0000', 'last_change': '2024-04-15T15:22:32.287169+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.287169+0000', 'last_undegraded': '2024-04-15T15:22:32.287169+0000', 'last_fullsized': '2024-04-15T15:22:32.287169+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2024-04-15T15:22:38.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.283134+0000', 'last_change': '2024-04-15T15:22:32.283134+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.283134+0000', 'last_undegraded': '2024-04-15T15:22:32.283134+0000', 'last_fullsized': '2024-04-15T15:22:32.283134+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-15T15:22:38.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-04-15T15:22:38.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.287033+0000', 'last_change': '2024-04-15T15:22:32.287033+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.287033+0000', 'last_undegraded': '2024-04-15T15:22:32.287033+0000', 'last_fullsized': '2024-04-15T15:22:32.287033+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:22:38.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2024-04-15T15:22:38.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.283165+0000', 'last_change': '2024-04-15T15:22:32.283165+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.283165+0000', 'last_undegraded': '2024-04-15T15:22:32.283165+0000', 'last_fullsized': '2024-04-15T15:22:32.283165+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.287661+0000', 'last_change': '2024-04-15T15:22:32.287661+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.287661+0000', 'last_undegraded': '2024-04-15T15:22:32.287661+0000', 'last_fullsized': '2024-04-15T15:22:32.287661+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282573+0000', 'last_change': '2024-04-15T15:22:32.282573+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282573+0000', 'last_undegraded': '2024-04-15T15:22:32.282573+0000', 'last_fullsized': '2024-04-15T15:22:32.282573+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.288692+0000', 'last_change': '2024-04-15T15:22:32.288692+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.288692+0000', 'last_undegraded': '2024-04-15T15:22:32.288692+0000', 'last_fullsized': '2024-04-15T15:22:32.288692+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.286380+0000', 'last_change': '2024-04-15T15:22:32.286380+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.286380+0000', 'last_undegraded': '2024-04-15T15:22:32.286380+0000', 'last_fullsized': '2024-04-15T15:22:32.286380+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.283560+0000', 'last_change': '2024-04-15T15:22:32.283560+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.283560+0000', 'last_undegraded': '2024-04-15T15:22:32.283560+0000', 'last_fullsized': '2024-04-15T15:22:32.283560+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2024-04-15T15:22:38.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.286540+0000', 'last_change': '2024-04-15T15:22:32.286540+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.286540+0000', 'last_undegraded': '2024-04-15T15:22:32.286540+0000', 'last_fullsized': '2024-04-15T15:22:32.286540+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282295+0000', 'last_change': '2024-04-15T15:22:32.282295+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282295+0000', 'last_undegraded': '2024-04-15T15:22:32.282295+0000', 'last_fullsized': '2024-04-15T15:22:32.282295+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.287436+0000', 'last_change': '2024-04-15T15:22:32.287436+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.287436+0000', 'last_undegraded': '2024-04-15T15:22:32.287436+0000', 'last_fullsized': '2024-04-15T15:22:32.287436+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.284239+0000', 'last_change': '2024-04-15T15:22:32.284239+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.284239+0000', 'last_undegraded': '2024-04-15T15:22:32.284239+0000', 'last_fullsized': '2024-04-15T15:22:32.284239+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.283378+0000', 'last_change': '2024-04-15T15:22:32.283378+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.283378+0000', 'last_undegraded': '2024-04-15T15:22:32.283378+0000', 'last_fullsized': '2024-04-15T15:22:32.283378+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.286533+0000', 'last_change': '2024-04-15T15:22:32.286533+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.286533+0000', 'last_undegraded': '2024-04-15T15:22:32.286533+0000', 'last_fullsized': '2024-04-15T15:22:32.286533+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.045 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.286417+0000', 'last_change': '2024-04-15T15:22:32.286417+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.286417+0000', 'last_undegraded': '2024-04-15T15:22:32.286417+0000', 'last_fullsized': '2024-04-15T15:22:32.286417+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.287816+0000', 'last_change': '2024-04-15T15:22:32.287816+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.287816+0000', 'last_undegraded': '2024-04-15T15:22:32.287816+0000', 'last_fullsized': '2024-04-15T15:22:32.287816+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.288081+0000', 'last_change': '2024-04-15T15:22:32.288081+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.288081+0000', 'last_undegraded': '2024-04-15T15:22:32.288081+0000', 'last_fullsized': '2024-04-15T15:22:32.288081+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.284349+0000', 'last_change': '2024-04-15T15:22:32.284349+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.284349+0000', 'last_undegraded': '2024-04-15T15:22:32.284349+0000', 'last_fullsized': '2024-04-15T15:22:32.284349+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.283316+0000', 'last_change': '2024-04-15T15:22:32.283316+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.283316+0000', 'last_undegraded': '2024-04-15T15:22:32.283316+0000', 'last_fullsized': '2024-04-15T15:22:32.283316+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-04-15T15:22:38.046 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282144+0000', 'last_change': '2024-04-15T15:22:32.282144+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282144+0000', 'last_undegraded': '2024-04-15T15:22:32.282144+0000', 'last_fullsized': '2024-04-15T15:22:32.282144+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.287768+0000', 'last_change': '2024-04-15T15:22:32.287768+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.287768+0000', 'last_undegraded': '2024-04-15T15:22:32.287768+0000', 'last_fullsized': '2024-04-15T15:22:32.287768+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282000+0000', 'last_change': '2024-04-15T15:22:32.282000+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282000+0000', 'last_undegraded': '2024-04-15T15:22:32.282000+0000', 'last_fullsized': '2024-04-15T15:22:32.282000+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282531+0000', 'last_change': '2024-04-15T15:22:32.282531+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282531+0000', 'last_undegraded': '2024-04-15T15:22:32.282531+0000', 'last_fullsized': '2024-04-15T15:22:32.282531+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.284589+0000', 'last_change': '2024-04-15T15:22:32.284589+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.284589+0000', 'last_undegraded': '2024-04-15T15:22:32.284589+0000', 'last_fullsized': '2024-04-15T15:22:32.284589+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-04-15T15:22:38.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.288577+0000', 'last_change': '2024-04-15T15:22:32.288577+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.288577+0000', 'last_undegraded': '2024-04-15T15:22:32.288577+0000', 'last_fullsized': '2024-04-15T15:22:32.288577+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-04-15T15:22:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.281997+0000', 'last_change': '2024-04-15T15:22:32.281997+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.281997+0000', 'last_undegraded': '2024-04-15T15:22:32.281997+0000', 'last_fullsized': '2024-04-15T15:22:32.281997+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:22:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-04-15T15:22:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.284953+0000', 'last_change': '2024-04-15T15:22:32.284953+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.284953+0000', 'last_undegraded': '2024-04-15T15:22:32.284953+0000', 'last_fullsized': '2024-04-15T15:22:32.284953+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-15T15:22:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-04-15T15:22:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.281544+0000', 'last_change': '2024-04-15T15:22:32.281544+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.281544+0000', 'last_undegraded': '2024-04-15T15:22:32.281544+0000', 'last_fullsized': '2024-04-15T15:22:32.281544+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:22:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-04-15T15:22:38.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.281381+0000', 'last_change': '2024-04-15T15:22:32.281381+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.281381+0000', 'last_undegraded': '2024-04-15T15:22:32.281381+0000', 'last_fullsized': '2024-04-15T15:22:32.281381+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.285849+0000', 'last_change': '2024-04-15T15:22:32.285849+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.285849+0000', 'last_undegraded': '2024-04-15T15:22:32.285849+0000', 'last_fullsized': '2024-04-15T15:22:32.285849+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282090+0000', 'last_change': '2024-04-15T15:22:32.282090+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282090+0000', 'last_undegraded': '2024-04-15T15:22:32.282090+0000', 'last_fullsized': '2024-04-15T15:22:32.282090+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.281983+0000', 'last_change': '2024-04-15T15:22:32.281983+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.281983+0000', 'last_undegraded': '2024-04-15T15:22:32.281983+0000', 'last_fullsized': '2024-04-15T15:22:32.281983+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.285787+0000', 'last_change': '2024-04-15T15:22:32.285787+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.285787+0000', 'last_undegraded': '2024-04-15T15:22:32.285787+0000', 'last_fullsized': '2024-04-15T15:22:32.285787+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.284873+0000', 'last_change': '2024-04-15T15:22:32.284873+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.284873+0000', 'last_undegraded': '2024-04-15T15:22:32.284873+0000', 'last_fullsized': '2024-04-15T15:22:32.284873+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.285847+0000', 'last_change': '2024-04-15T15:22:32.285847+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.285847+0000', 'last_undegraded': '2024-04-15T15:22:32.285847+0000', 'last_fullsized': '2024-04-15T15:22:32.285847+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.285710+0000', 'last_change': '2024-04-15T15:22:32.285710+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.285710+0000', 'last_undegraded': '2024-04-15T15:22:32.285710+0000', 'last_fullsized': '2024-04-15T15:22:32.285710+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.280827+0000', 'last_change': '2024-04-15T15:22:32.280827+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.280827+0000', 'last_undegraded': '2024-04-15T15:22:32.280827+0000', 'last_fullsized': '2024-04-15T15:22:32.280827+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.284940+0000', 'last_change': '2024-04-15T15:22:32.284940+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.284940+0000', 'last_undegraded': '2024-04-15T15:22:32.284940+0000', 'last_fullsized': '2024-04-15T15:22:32.284940+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-04-15T15:22:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.281646+0000', 'last_change': '2024-04-15T15:22:32.281646+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.281646+0000', 'last_undegraded': '2024-04-15T15:22:32.281646+0000', 'last_fullsized': '2024-04-15T15:22:32.281646+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-04-15T15:22:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.285229+0000', 'last_change': '2024-04-15T15:22:32.285229+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.285229+0000', 'last_undegraded': '2024-04-15T15:22:32.285229+0000', 'last_fullsized': '2024-04-15T15:22:32.285229+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:22:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-04-15T15:22:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.288112+0000', 'last_change': '2024-04-15T15:22:32.288112+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.288112+0000', 'last_undegraded': '2024-04-15T15:22:32.288112+0000', 'last_fullsized': '2024-04-15T15:22:32.288112+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-04-15T15:22:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.282394+0000', 'last_change': '2024-04-15T15:22:32.282394+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.282394+0000', 'last_undegraded': '2024-04-15T15:22:32.282394+0000', 'last_fullsized': '2024-04-15T15:22:32.282394+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:22:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-04-15T15:22:38.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.280827+0000', 'last_change': '2024-04-15T15:22:32.280827+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.280827+0000', 'last_undegraded': '2024-04-15T15:22:32.280827+0000', 'last_fullsized': '2024-04-15T15:22:32.280827+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:22:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-04-15T15:22:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.286818+0000', 'last_change': '2024-04-15T15:22:32.286818+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.286818+0000', 'last_undegraded': '2024-04-15T15:22:32.286818+0000', 'last_fullsized': '2024-04-15T15:22:32.286818+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:22:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-04-15T15:22:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.283018+0000', 'last_change': '2024-04-15T15:22:32.283018+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.283018+0000', 'last_undegraded': '2024-04-15T15:22:32.283018+0000', 'last_fullsized': '2024-04-15T15:22:32.283018+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-15T15:22:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-04-15T15:22:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.287505+0000', 'last_change': '2024-04-15T15:22:32.287505+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.287505+0000', 'last_undegraded': '2024-04-15T15:22:32.287505+0000', 'last_fullsized': '2024-04-15T15:22:32.287505+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:22:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-04-15T15:22:38.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:22:32.285960+0000', 'last_change': '2024-04-15T15:22:32.285960+0000', 'last_active': '2024-04-15T15:22:32.266162+0000', 'last_peered': '2024-04-15T15:22:32.266162+0000', 'last_clean': '2024-04-15T15:22:32.266162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:22:32.285960+0000', 'last_undegraded': '2024-04-15T15:22:32.285960+0000', 'last_fullsized': '2024-04-15T15:22:32.285960+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'last_clean_scrub_stamp': '2024-04-15T15:22:32.266162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T15:22:38.053 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-15T15:22:38.079 INFO:tasks.ceph.mds.d.smithi055.stderr:2024-04-15T15:22:38.078+0000 7f63f981b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:22:38.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.177+0000 7f901d293640 1 Processor -- start 2024-04-15T15:22:38.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.178+0000 7f901d293640 1 -- start start 2024-04-15T15:22:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.178+0000 7f901d293640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f90180d1770 0x7f90180d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.178+0000 7f901d293640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90180d5190 con 0x7f9018161fe0 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.178+0000 7f901d293640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90180cf3c0 con 0x7f90180b35f0 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.178+0000 7f901d293640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f90180cf6a0 con 0x7f90180d1770 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.178+0000 7f9016ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9018161fe0 0x7f90180b3fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35962/0 (socket says 172.21.15.12:35962) 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.178+0000 7f9016ffd640 1 -- 172.21.15.12:0/2175583258 learned_addr learned my addr 172.21.15.12:0/2175583258 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f90177fe640 1 --2- 172.21.15.12:0/2175583258 >> v2:172.21.15.55:3300/0 conn(0x7f90180d1770 0x7f90180d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057432977 0 0) 0x7f90180d5190 con 0x7f9018161fe0 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9000003520 con 0x7f9018161fe0 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3170824123 0 0) 0x7f9000003520 con 0x7f9018161fe0 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 >> v2:172.21.15.55:3300/0 conn(0x7f90180d1770 msgr2=0x7f90180d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 --2- 172.21.15.12:0/2175583258 >> v2:172.21.15.55:3300/0 conn(0x7f90180d1770 0x7f90180d3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 >> v1:172.21.15.55:6789/0 conn(0x7f90180b35f0 legacy=0x7f90180d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90180cf980 con 0x7f9018161fe0 2024-04-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f9008001670 con 0x7f9018161fe0 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f90177fe640 1 --2- 172.21.15.12:0/2175583258 >> v2:172.21.15.55:3300/0 conn(0x7f90180d1770 0x7f90180d3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 >> v1:172.21.15.12:6789/0 conn(0x7f9018161fe0 legacy=0x7f90180b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90000039b0 0x7f9000003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9000004800 0x7f9000004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90000051c0 0x7f900000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90180cf6a0 con 0x7f90000039b0 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f900000c060 con 0x7f9000004800 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f900000c260 con 0x7f90000051c0 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9016ffd640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90000039b0 0x7f9000003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f90167fc640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9000004800 0x7f9000004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f90177fe640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90000051c0 0x7f900000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9016ffd640 1 -- 172.21.15.12:0/2175583258 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90000051c0 msgr2=0x7f900000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9016ffd640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90000051c0 0x7f900000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9016ffd640 1 -- 172.21.15.12:0/2175583258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9000004800 msgr2=0x7f9000004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9016ffd640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9000004800 0x7f9000004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f9016ffd640 1 -- 172.21.15.12:0/2175583258 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9000003520 con 0x7f90000039b0 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.179+0000 7f90177fe640 1 -- 172.21.15.12:0/2175583258 reap_dead start 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.180+0000 7f9016ffd640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90000039b0 0x7f9000003dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f900800c240 tx=0x7f9000005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73ac34f48f190be server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.180+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9008007cf0 con 0x7f90000039b0 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.180+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9008030070 con 0x7f90000039b0 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.180+0000 7f9015ffb640 1 -- 172.21.15.12:0/2175583258 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9008005d70 con 0x7f90000039b0 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.181+0000 7f901d293640 1 -- 172.21.15.12:0/2175583258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90000039b0 msgr2=0x7f9000003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.181+0000 7f901d293640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90000039b0 0x7f9000003dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f900800c240 tx=0x7f9000005830 comp rx=0 tx=0).stop 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.181+0000 7f901d293640 1 -- 172.21.15.12:0/2175583258 shutdown_connections 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.181+0000 7f901d293640 1 --2- 172.21.15.12:0/2175583258 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90000039b0 0x7f9000003dd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.181+0000 7f901d293640 1 -- 172.21.15.12:0/2175583258 >> 172.21.15.12:0/2175583258 conn(0x7f901808bf60 msgr2=0x7f90180d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.181+0000 7f901d293640 1 -- 172.21.15.12:0/2175583258 shutdown_connections 2024-04-15T15:22:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.181+0000 7f901d293640 1 -- 172.21.15.12:0/2175583258 wait complete. 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.182+0000 7f901d293640 1 Processor -- start 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.182+0000 7f901d293640 1 -- start start 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f901d293640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90000039b0 0x7f901815ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f901d293640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90180c97d0 0x7f90180cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f901d293640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90180ce3f0 0x7f901815c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f901d293640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90180d4d60 con 0x7f90180c97d0 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f901d293640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f90181623c0 con 0x7f90000039b0 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f901d293640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f90180d4ee0 con 0x7f90180ce3f0 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90167fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90180c97d0 0x7f90180cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90167fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90180c97d0 0x7f90180cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49546/0 (socket says 172.21.15.12:49546) 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90167fc640 1 -- 172.21.15.12:0/286298319 learned_addr learned my addr 172.21.15.12:0/286298319 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90177fe640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90180ce3f0 0x7f901815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f9016ffd640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90000039b0 0x7f901815ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90167fc640 1 -- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90180ce3f0 msgr2=0x7f901815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90167fc640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90180ce3f0 0x7f901815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90167fc640 1 -- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90000039b0 msgr2=0x7f901815ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90167fc640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90000039b0 0x7f901815ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90167fc640 1 -- 172.21.15.12:0/286298319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90180c8730 con 0x7f90180c97d0 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f9016ffd640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90000039b0 0x7f901815ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.183+0000 7f90177fe640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90180ce3f0 0x7f901815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.184+0000 7f90167fc640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90180c97d0 0x7f90180cdeb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f900c00f5f0 tx=0x7f900c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.184+0000 7f8fff7fe640 1 -- 172.21.15.12:0/286298319 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900c0052e0 con 0x7f90180c97d0 2024-04-15T15:22:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.184+0000 7f8fff7fe640 1 -- 172.21.15.12:0/286298319 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f900c018070 con 0x7f90180c97d0 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.184+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90180c8a10 con 0x7f90180c97d0 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.184+0000 7f8fff7fe640 1 -- 172.21.15.12:0/286298319 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900c003810 con 0x7f90180c97d0 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.184+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90180c8fd0 con 0x7f90180c97d0 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.185+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90180689e0 con 0x7f90180c97d0 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.185+0000 7f8fff7fe640 1 -- 172.21.15.12:0/286298319 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f900c0039b0 con 0x7f90180c97d0 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.186+0000 7f8fff7fe640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f900406c8f0 0x7f900406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.186+0000 7f8fff7fe640 1 -- 172.21.15.12:0/286298319 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f900c08ba30 con 0x7f90180c97d0 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.186+0000 7f9016ffd640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f900406c8f0 0x7f900406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.186+0000 7f9016ffd640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f900406c8f0 0x7f900406edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9008034000 tx=0x7f9008039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.190+0000 7f8fff7fe640 1 -- 172.21.15.12:0/286298319 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f900c059d60 con 0x7f90180c97d0 2024-04-15T15:22:38.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.449+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f901808e100 con 0x7f90180c97d0 2024-04-15T15:22:38.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.449+0000 7f8fff7fe640 1 -- 172.21.15.12:0/286298319 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1034 (secure 0 0 0) 0x7f900c059660 con 0x7f90180c97d0 2024-04-15T15:22:38.452 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:38.452 INFO:teuthology.orchestra.run.smithi012.stdout:{"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":112,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1713194457,"num_in_osds":8,"osd_in_since":1713194448,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":242909184,"bytes_avail":772851204096,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":18,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5203}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-15T15:22:16.604028+0000","services":{}},"progress_events":{}} 2024-04-15T15:22:38.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f900406c8f0 msgr2=0x7f900406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f900406c8f0 0x7f900406edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9008034000 tx=0x7f9008039040 comp rx=0 tx=0).stop 2024-04-15T15:22:38.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90180c97d0 msgr2=0x7f90180cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90180c97d0 0x7f90180cdeb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f900c00f5f0 tx=0x7f900c00fab0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 shutdown_connections 2024-04-15T15:22:38.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f90180ce3f0 0x7f901815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90180c97d0 0x7f90180cdeb0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f900406c8f0 0x7f900406edb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 --2- 172.21.15.12:0/286298319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90000039b0 0x7f901815ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 >> 172.21.15.12:0/286298319 conn(0x7f901808bf60 msgr2=0x7f90180b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:38.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 shutdown_connections 2024-04-15T15:22:38.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.451+0000 7f901d293640 1 -- 172.21.15.12:0/286298319 wait complete. 2024-04-15T15:22:38.466 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-15T15:22:38.466 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:22:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.569+0000 7f8f0f69a640 1 Processor -- start 2024-04-15T15:22:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.569+0000 7f8f0f69a640 1 -- start start 2024-04-15T15:22:38.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0f69a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8f0809a9d0 0x7f8f08160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0f69a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f080701f0 0x7f8f0815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0f69a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809adb0 0x7f8f080d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0f69a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8f0809ba90 con 0x7f8f080701f0 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0f69a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8f08160070 con 0x7f8f0809a9d0 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0f69a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8f080d08c0 con 0x7f8f0809adb0 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0cc35640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f080701f0 0x7f8f0815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0cc35640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f080701f0 0x7f8f0815fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49562/0 (socket says 172.21.15.12:49562) 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0cc35640 1 -- 172.21.15.12:0/1377572014 learned_addr learned my addr 172.21.15.12:0/1377572014 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0cc35640 1 -- 172.21.15.12:0/1377572014 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809adb0 msgr2=0x7f8f080d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0d436640 1 --2- 172.21.15.12:0/1377572014 >> v2:172.21.15.55:3300/0 conn(0x7f8f0809a9d0 0x7f8f08160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0dc37640 1 --2- 172.21.15.12:0/1377572014 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809adb0 0x7f8f080d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0cc35640 1 --2- 172.21.15.12:0/1377572014 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809adb0 0x7f8f080d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0cc35640 1 -- 172.21.15.12:0/1377572014 >> v2:172.21.15.55:3300/0 conn(0x7f8f0809a9d0 msgr2=0x7f8f08160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0cc35640 1 --2- 172.21.15.12:0/1377572014 >> v2:172.21.15.55:3300/0 conn(0x7f8f0809a9d0 0x7f8f08160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.570+0000 7f8f0cc35640 1 -- 172.21.15.12:0/1377572014 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f08070720 con 0x7f8f080701f0 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.571+0000 7f8f0cc35640 1 --2- 172.21.15.12:0/1377572014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f0815fb30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8f080d30a0 tx=0x7f8efc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce3169e4e386712 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.571+0000 7f8f07fff640 1 -- 172.21.15.12:0/1377572014 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc035020 con 0x7f8f080701f0 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.571+0000 7f8f07fff640 1 -- 172.21.15.12:0/1377572014 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8efc031640 con 0x7f8f080701f0 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.571+0000 7f8f07fff640 1 -- 172.21.15.12:0/1377572014 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc02d040 con 0x7f8f080701f0 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.571+0000 7f8f0f69a640 1 -- 172.21.15.12:0/1377572014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 msgr2=0x7f8f0815fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.571+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/1377572014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f0815fb30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8f080d30a0 tx=0x7f8efc02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.572+0000 7f8f0f69a640 1 -- 172.21.15.12:0/1377572014 shutdown_connections 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.572+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/1377572014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f0815fb30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.572+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/1377572014 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809adb0 0x7f8f080d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.572+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/1377572014 >> v2:172.21.15.55:3300/0 conn(0x7f8f0809a9d0 0x7f8f08160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.572+0000 7f8f0f69a640 1 -- 172.21.15.12:0/1377572014 >> 172.21.15.12:0/1377572014 conn(0x7f8f080d2460 msgr2=0x7f8f080d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.572+0000 7f8f0f69a640 1 -- 172.21.15.12:0/1377572014 shutdown_connections 2024-04-15T15:22:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.572+0000 7f8f0f69a640 1 -- 172.21.15.12:0/1377572014 wait complete. 2024-04-15T15:22:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.573+0000 7f8f0f69a640 1 Processor -- start 2024-04-15T15:22:38.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.573+0000 7f8f0f69a640 1 -- start start 2024-04-15T15:22:38.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.573+0000 7f8f0f69a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f08147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.573+0000 7f8f0f69a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809a9d0 0x7f8f0815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.573+0000 7f8f0f69a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f0809adb0 0x7f8f080cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0f69a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f08071050 con 0x7f8f080701f0 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0f69a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8f08070d70 con 0x7f8f0809adb0 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0f69a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8f080b1c60 con 0x7f8f0809a9d0 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f08147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f08147f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49576/0 (socket says 172.21.15.12:49576) 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 -- 172.21.15.12:0/469257091 learned_addr learned my addr 172.21.15.12:0/469257091 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 -- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809a9d0 msgr2=0x7f8f0815c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809a9d0 0x7f8f0815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 -- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f0809adb0 msgr2=0x7f8f080cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0dc37640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f0809adb0 0x7f8f080cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f0809adb0 0x7f8f080cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 -- 172.21.15.12:0/469257091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0815d2e0 con 0x7f8f080701f0 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0dc37640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f0809adb0 0x7f8f080cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:38.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0d436640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f08147f70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800c7f0 tx=0x7f8ef800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f05ffb640 1 -- 172.21.15.12:0/469257091 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8015070 con 0x7f8f080701f0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f05ffb640 1 -- 172.21.15.12:0/469257091 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ef8004220 con 0x7f8f080701f0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f05ffb640 1 -- 172.21.15.12:0/469257091 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8007620 con 0x7f8f080701f0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.574+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0815cd20 con 0x7f8f080701f0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.575+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f08148ad0 con 0x7f8f080701f0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.575+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f080689e0 con 0x7f8f080701f0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.576+0000 7f8f05ffb640 1 -- 172.21.15.12:0/469257091 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ef80079d0 con 0x7f8f080701f0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.576+0000 7f8f05ffb640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8ef406c8f0 0x7f8ef406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.576+0000 7f8f05ffb640 1 -- 172.21.15.12:0/469257091 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8ef808bdf0 con 0x7f8f080701f0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.576+0000 7f8f0cc35640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8ef406c8f0 0x7f8ef406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:38.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.577+0000 7f8f0cc35640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8ef406c8f0 0x7f8ef406edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8efc036000 tx=0x7f8efc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:38.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.581+0000 7f8f05ffb640 1 -- 172.21.15.12:0/469257091 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ef805a000 con 0x7f8f080701f0 2024-04-15T15:22:38.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.762+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8f080d3370 con 0x7f8ef406c8f0 2024-04-15T15:22:38.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.769+0000 7f8f05ffb640 1 -- 172.21.15.12:0/469257091 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251959 (secure 0 0 0) 0x7f8f080d3370 con 0x7f8ef406c8f0 2024-04-15T15:22:38.775 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:38.776 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:22:38.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8ef406c8f0 msgr2=0x7f8ef406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8ef406c8f0 0x7f8ef406edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8efc036000 tx=0x7f8efc030000 comp rx=0 tx=0).stop 2024-04-15T15:22:38.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 msgr2=0x7f8f08147f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:38.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f08147f70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800c7f0 tx=0x7f8ef800ccb0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 shutdown_connections 2024-04-15T15:22:38.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8ef406c8f0 0x7f8ef406edb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f0809adb0 0x7f8f080cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f0809a9d0 0x7f8f0815c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 --2- 172.21.15.12:0/469257091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f080701f0 0x7f8f08147f70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:38.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 >> 172.21.15.12:0/469257091 conn(0x7f8f080d2460 msgr2=0x7f8f0806d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:38.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 shutdown_connections 2024-04-15T15:22:38.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:38.772+0000 7f8f0f69a640 1 -- 172.21.15.12:0/469257091 wait complete. 2024-04-15T15:22:38.785 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-15T15:22:38.608842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237584,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737136,"statfs":{"total":773094113280,"available":772850827264,"internally_reserved":0,"allocated":4857856,"data_stored":2745804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.330431"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295604+0000","last_change":"2024-04-15T15:22:33.295313+0000","last_active":"2024-04-15T15:22:34.295604+0000","last_peered":"2024-04-15T15:22:34.295604+0000","last_clean":"2024-04-15T15:22:34.295604+0000","last_became_active":"2024-04-15T15:22:33.295019+0000","last_became_peered":"2024-04-15T15:22:33.295019+0000","last_unstale":"2024-04-15T15:22:34.295604+0000","last_undegraded":"2024-04-15T15:22:34.295604+0000","last_fullsized":"2024-04-15T15:22:34.295604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:46:17.698031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295342+0000","last_change":"2024-04-15T15:22:33.294677+0000","last_active":"2024-04-15T15:22:34.295342+0000","last_peered":"2024-04-15T15:22:34.295342+0000","last_clean":"2024-04-15T15:22:34.295342+0000","last_became_active":"2024-04-15T15:22:33.294383+0000","last_became_peered":"2024-04-15T15:22:33.294383+0000","last_unstale":"2024-04-15T15:22:34.295342+0000","last_undegraded":"2024-04-15T15:22:34.295342+0000","last_fullsized":"2024-04-15T15:22:34.295342+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:47.415134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565304+0000","last_change":"2024-04-15T15:22:33.308305+0000","last_active":"2024-04-15T15:22:34.565304+0000","last_peered":"2024-04-15T15:22:34.565304+0000","last_clean":"2024-04-15T15:22:34.565304+0000","last_became_active":"2024-04-15T15:22:33.308000+0000","last_became_peered":"2024-04-15T15:22:33.308000+0000","last_unstale":"2024-04-15T15:22:34.565304+0000","last_undegraded":"2024-04-15T15:22:34.565304+0000","last_fullsized":"2024-04-15T15:22:34.565304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:30:58.162485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039672+0000","last_change":"2024-04-15T15:22:33.305255+0000","last_active":"2024-04-15T15:22:35.039672+0000","last_peered":"2024-04-15T15:22:35.039672+0000","last_clean":"2024-04-15T15:22:35.039672+0000","last_became_active":"2024-04-15T15:22:33.304994+0000","last_became_peered":"2024-04-15T15:22:33.304994+0000","last_unstale":"2024-04-15T15:22:35.039672+0000","last_undegraded":"2024-04-15T15:22:35.039672+0000","last_fullsized":"2024-04-15T15:22:35.039672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:28.120264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039279+0000","last_change":"2024-04-15T15:22:33.304553+0000","last_active":"2024-04-15T15:22:35.039279+0000","last_peered":"2024-04-15T15:22:35.039279+0000","last_clean":"2024-04-15T15:22:35.039279+0000","last_became_active":"2024-04-15T15:22:33.304200+0000","last_became_peered":"2024-04-15T15:22:33.304200+0000","last_unstale":"2024-04-15T15:22:35.039279+0000","last_undegraded":"2024-04-15T15:22:35.039279+0000","last_fullsized":"2024-04-15T15:22:35.039279+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:48.069224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039099+0000","last_change":"2024-04-15T15:22:33.296731+0000","last_active":"2024-04-15T15:22:35.039099+0000","last_peered":"2024-04-15T15:22:35.039099+0000","last_clean":"2024-04-15T15:22:35.039099+0000","last_became_active":"2024-04-15T15:22:33.295992+0000","last_became_peered":"2024-04-15T15:22:33.295992+0000","last_unstale":"2024-04-15T15:22:35.039099+0000","last_undegraded":"2024-04-15T15:22:35.039099+0000","last_fullsized":"2024-04-15T15:22:35.039099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:25:09.987700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305511+0000","last_change":"2024-04-15T15:22:33.298931+0000","last_active":"2024-04-15T15:22:34.305511+0000","last_peered":"2024-04-15T15:22:34.305511+0000","last_clean":"2024-04-15T15:22:34.305511+0000","last_became_active":"2024-04-15T15:22:33.298626+0000","last_became_peered":"2024-04-15T15:22:33.298626+0000","last_unstale":"2024-04-15T15:22:34.305511+0000","last_undegraded":"2024-04-15T15:22:34.305511+0000","last_fullsized":"2024-04-15T15:22:34.305511+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:33.588528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295124+0000","last_change":"2024-04-15T15:22:33.307911+0000","last_active":"2024-04-15T15:22:34.295124+0000","last_peered":"2024-04-15T15:22:34.295124+0000","last_clean":"2024-04-15T15:22:34.295124+0000","last_became_active":"2024-04-15T15:22:33.307643+0000","last_became_peered":"2024-04-15T15:22:33.307643+0000","last_unstale":"2024-04-15T15:22:34.295124+0000","last_undegraded":"2024-04-15T15:22:34.295124+0000","last_fullsized":"2024-04-15T15:22:34.295124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:11:33.617468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565545+0000","last_change":"2024-04-15T15:22:33.313825+0000","last_active":"2024-04-15T15:22:34.565545+0000","last_peered":"2024-04-15T15:22:34.565545+0000","last_clean":"2024-04-15T15:22:34.565545+0000","last_became_active":"2024-04-15T15:22:33.313619+0000","last_became_peered":"2024-04-15T15:22:33.313619+0000","last_unstale":"2024-04-15T15:22:34.565545+0000","last_undegraded":"2024-04-15T15:22:34.565545+0000","last_fullsized":"2024-04-15T15:22:34.565545+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:53:19.613211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305991+0000","last_change":"2024-04-15T15:22:33.314412+0000","last_active":"2024-04-15T15:22:34.305991+0000","last_peered":"2024-04-15T15:22:34.305991+0000","last_clean":"2024-04-15T15:22:34.305991+0000","last_became_active":"2024-04-15T15:22:33.313558+0000","last_became_peered":"2024-04-15T15:22:33.313558+0000","last_unstale":"2024-04-15T15:22:34.305991+0000","last_undegraded":"2024-04-15T15:22:34.305991+0000","last_fullsized":"2024-04-15T15:22:34.305991+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:26:40.045991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306181+0000","last_change":"2024-04-15T15:22:33.306879+0000","last_active":"2024-04-15T15:22:34.306181+0000","last_peered":"2024-04-15T15:22:34.306181+0000","last_clean":"2024-04-15T15:22:34.306181+0000","last_became_active":"2024-04-15T15:22:33.306416+0000","last_became_peered":"2024-04-15T15:22:33.306416+0000","last_unstale":"2024-04-15T15:22:34.306181+0000","last_undegraded":"2024-04-15T15:22:34.306181+0000","last_fullsized":"2024-04-15T15:22:34.306181+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:52.844126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565399+0000","last_change":"2024-04-15T15:22:33.298571+0000","last_active":"2024-04-15T15:22:34.565399+0000","last_peered":"2024-04-15T15:22:34.565399+0000","last_clean":"2024-04-15T15:22:34.565399+0000","last_became_active":"2024-04-15T15:22:33.298254+0000","last_became_peered":"2024-04-15T15:22:33.298254+0000","last_unstale":"2024-04-15T15:22:34.565399+0000","last_undegraded":"2024-04-15T15:22:34.565399+0000","last_fullsized":"2024-04-15T15:22:34.565399+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:47:45.911804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305719+0000","last_change":"2024-04-15T15:22:33.313702+0000","last_active":"2024-04-15T15:22:34.305719+0000","last_peered":"2024-04-15T15:22:34.305719+0000","last_clean":"2024-04-15T15:22:34.305719+0000","last_became_active":"2024-04-15T15:22:33.313391+0000","last_became_peered":"2024-04-15T15:22:33.313391+0000","last_unstale":"2024-04-15T15:22:34.305719+0000","last_undegraded":"2024-04-15T15:22:34.305719+0000","last_fullsized":"2024-04-15T15:22:34.305719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:47.415134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538077+0000","last_change":"2024-04-15T15:22:33.301615+0000","last_active":"2024-04-15T15:22:34.538077+0000","last_peered":"2024-04-15T15:22:34.538077+0000","last_clean":"2024-04-15T15:22:34.538077+0000","last_became_active":"2024-04-15T15:22:33.301298+0000","last_became_peered":"2024-04-15T15:22:33.301298+0000","last_unstale":"2024-04-15T15:22:34.538077+0000","last_undegraded":"2024-04-15T15:22:34.538077+0000","last_fullsized":"2024-04-15T15:22:34.538077+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:53:39.518417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305528+0000","last_change":"2024-04-15T15:22:33.300507+0000","last_active":"2024-04-15T15:22:34.305528+0000","last_peered":"2024-04-15T15:22:34.305528+0000","last_clean":"2024-04-15T15:22:34.305528+0000","last_became_active":"2024-04-15T15:22:33.300105+0000","last_became_peered":"2024-04-15T15:22:33.300105+0000","last_unstale":"2024-04-15T15:22:34.305528+0000","last_undegraded":"2024-04-15T15:22:34.305528+0000","last_fullsized":"2024-04-15T15:22:34.305528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:26.586364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705033+0000","last_change":"2024-04-15T15:22:33.314263+0000","last_active":"2024-04-15T15:22:34.705033+0000","last_peered":"2024-04-15T15:22:34.705033+0000","last_clean":"2024-04-15T15:22:34.705033+0000","last_became_active":"2024-04-15T15:22:33.313829+0000","last_became_peered":"2024-04-15T15:22:33.313829+0000","last_unstale":"2024-04-15T15:22:34.705033+0000","last_undegraded":"2024-04-15T15:22:34.705033+0000","last_fullsized":"2024-04-15T15:22:34.705033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:51:51.191240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705498+0000","last_change":"2024-04-15T15:22:33.314419+0000","last_active":"2024-04-15T15:22:34.705498+0000","last_peered":"2024-04-15T15:22:34.705498+0000","last_clean":"2024-04-15T15:22:34.705498+0000","last_became_active":"2024-04-15T15:22:33.313923+0000","last_became_peered":"2024-04-15T15:22:33.313923+0000","last_unstale":"2024-04-15T15:22:34.705498+0000","last_undegraded":"2024-04-15T15:22:34.705498+0000","last_fullsized":"2024-04-15T15:22:34.705498+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:47.415134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038948+0000","last_change":"2024-04-15T15:22:33.305542+0000","last_active":"2024-04-15T15:22:35.038948+0000","last_peered":"2024-04-15T15:22:35.038948+0000","last_clean":"2024-04-15T15:22:35.038948+0000","last_became_active":"2024-04-15T15:22:33.305231+0000","last_became_peered":"2024-04-15T15:22:33.305231+0000","last_unstale":"2024-04-15T15:22:35.038948+0000","last_undegraded":"2024-04-15T15:22:35.038948+0000","last_fullsized":"2024-04-15T15:22:35.038948+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:18:35.877536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039548+0000","last_change":"2024-04-15T15:22:33.302274+0000","last_active":"2024-04-15T15:22:35.039548+0000","last_peered":"2024-04-15T15:22:35.039548+0000","last_clean":"2024-04-15T15:22:35.039548+0000","last_became_active":"2024-04-15T15:22:33.301981+0000","last_became_peered":"2024-04-15T15:22:33.301981+0000","last_unstale":"2024-04-15T15:22:35.039548+0000","last_undegraded":"2024-04-15T15:22:35.039548+0000","last_fullsized":"2024-04-15T15:22:35.039548+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:21:14.004136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038970+0000","last_change":"2024-04-15T15:22:33.305686+0000","last_active":"2024-04-15T15:22:35.038970+0000","last_peered":"2024-04-15T15:22:35.038970+0000","last_clean":"2024-04-15T15:22:35.038970+0000","last_became_active":"2024-04-15T15:22:33.305390+0000","last_became_peered":"2024-04-15T15:22:33.305390+0000","last_unstale":"2024-04-15T15:22:35.038970+0000","last_undegraded":"2024-04-15T15:22:35.038970+0000","last_fullsized":"2024-04-15T15:22:35.038970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:53:39.518417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305803+0000","last_change":"2024-04-15T15:22:33.301798+0000","last_active":"2024-04-15T15:22:34.305803+0000","last_peered":"2024-04-15T15:22:34.305803+0000","last_clean":"2024-04-15T15:22:34.305803+0000","last_became_active":"2024-04-15T15:22:33.301513+0000","last_became_peered":"2024-04-15T15:22:33.301513+0000","last_unstale":"2024-04-15T15:22:34.305803+0000","last_undegraded":"2024-04-15T15:22:34.305803+0000","last_fullsized":"2024-04-15T15:22:34.305803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:51:51.191240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538355+0000","last_change":"2024-04-15T15:22:33.302535+0000","last_active":"2024-04-15T15:22:34.538355+0000","last_peered":"2024-04-15T15:22:34.538355+0000","last_clean":"2024-04-15T15:22:34.538355+0000","last_became_active":"2024-04-15T15:22:33.301667+0000","last_became_peered":"2024-04-15T15:22:33.301667+0000","last_unstale":"2024-04-15T15:22:34.538355+0000","last_undegraded":"2024-04-15T15:22:34.538355+0000","last_fullsized":"2024-04-15T15:22:34.538355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:18:22.291418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705072+0000","last_change":"2024-04-15T15:22:33.308787+0000","last_active":"2024-04-15T15:22:34.705072+0000","last_peered":"2024-04-15T15:22:34.705072+0000","last_clean":"2024-04-15T15:22:34.705072+0000","last_became_active":"2024-04-15T15:22:33.308238+0000","last_became_peered":"2024-04-15T15:22:33.308238+0000","last_unstale":"2024-04-15T15:22:34.705072+0000","last_undegraded":"2024-04-15T15:22:34.705072+0000","last_fullsized":"2024-04-15T15:22:34.705072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:22:11.084708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039850+0000","last_change":"2024-04-15T15:22:33.304264+0000","last_active":"2024-04-15T15:22:35.039850+0000","last_peered":"2024-04-15T15:22:35.039850+0000","last_clean":"2024-04-15T15:22:35.039850+0000","last_became_active":"2024-04-15T15:22:33.303941+0000","last_became_peered":"2024-04-15T15:22:33.303941+0000","last_unstale":"2024-04-15T15:22:35.039850+0000","last_undegraded":"2024-04-15T15:22:35.039850+0000","last_fullsized":"2024-04-15T15:22:35.039850+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:02:15.691472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038908+0000","last_change":"2024-04-15T15:22:33.297937+0000","last_active":"2024-04-15T15:22:35.038908+0000","last_peered":"2024-04-15T15:22:35.038908+0000","last_clean":"2024-04-15T15:22:35.038908+0000","last_became_active":"2024-04-15T15:22:33.297425+0000","last_became_peered":"2024-04-15T15:22:33.297425+0000","last_unstale":"2024-04-15T15:22:35.038908+0000","last_undegraded":"2024-04-15T15:22:35.038908+0000","last_fullsized":"2024-04-15T15:22:35.038908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:17:29.189541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039595+0000","last_change":"2024-04-15T15:22:33.301393+0000","last_active":"2024-04-15T15:22:35.039595+0000","last_peered":"2024-04-15T15:22:35.039595+0000","last_clean":"2024-04-15T15:22:35.039595+0000","last_became_active":"2024-04-15T15:22:33.299906+0000","last_became_peered":"2024-04-15T15:22:33.299906+0000","last_unstale":"2024-04-15T15:22:35.039595+0000","last_undegraded":"2024-04-15T15:22:35.039595+0000","last_fullsized":"2024-04-15T15:22:35.039595+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:30:58.162485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038906+0000","last_change":"2024-04-15T15:22:33.298932+0000","last_active":"2024-04-15T15:22:35.038906+0000","last_peered":"2024-04-15T15:22:35.038906+0000","last_clean":"2024-04-15T15:22:35.038906+0000","last_became_active":"2024-04-15T15:22:33.298633+0000","last_became_peered":"2024-04-15T15:22:33.298633+0000","last_unstale":"2024-04-15T15:22:35.038906+0000","last_undegraded":"2024-04-15T15:22:35.038906+0000","last_fullsized":"2024-04-15T15:22:35.038906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:02:15.691472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305804+0000","last_change":"2024-04-15T15:22:33.302161+0000","last_active":"2024-04-15T15:22:34.305804+0000","last_peered":"2024-04-15T15:22:34.305804+0000","last_clean":"2024-04-15T15:22:34.305804+0000","last_became_active":"2024-04-15T15:22:33.299226+0000","last_became_peered":"2024-04-15T15:22:33.299226+0000","last_unstale":"2024-04-15T15:22:34.305804+0000","last_undegraded":"2024-04-15T15:22:34.305804+0000","last_fullsized":"2024-04-15T15:22:34.305804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:18.339496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038719+0000","last_change":"2024-04-15T15:22:33.300377+0000","last_active":"2024-04-15T15:22:35.038719+0000","last_peered":"2024-04-15T15:22:35.038719+0000","last_clean":"2024-04-15T15:22:35.038719+0000","last_became_active":"2024-04-15T15:22:33.300088+0000","last_became_peered":"2024-04-15T15:22:33.300088+0000","last_unstale":"2024-04-15T15:22:35.038719+0000","last_undegraded":"2024-04-15T15:22:35.038719+0000","last_fullsized":"2024-04-15T15:22:35.038719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:46.712200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038706+0000","last_change":"2024-04-15T15:22:33.298247+0000","last_active":"2024-04-15T15:22:35.038706+0000","last_peered":"2024-04-15T15:22:35.038706+0000","last_clean":"2024-04-15T15:22:35.038706+0000","last_became_active":"2024-04-15T15:22:33.297608+0000","last_became_peered":"2024-04-15T15:22:33.297608+0000","last_unstale":"2024-04-15T15:22:35.038706+0000","last_undegraded":"2024-04-15T15:22:35.038706+0000","last_fullsized":"2024-04-15T15:22:35.038706+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:48.069224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705123+0000","last_change":"2024-04-15T15:22:33.308795+0000","last_active":"2024-04-15T15:22:34.705123+0000","last_peered":"2024-04-15T15:22:34.705123+0000","last_clean":"2024-04-15T15:22:34.705123+0000","last_became_active":"2024-04-15T15:22:33.308527+0000","last_became_peered":"2024-04-15T15:22:33.308527+0000","last_unstale":"2024-04-15T15:22:34.705123+0000","last_undegraded":"2024-04-15T15:22:34.705123+0000","last_fullsized":"2024-04-15T15:22:34.705123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:53:19.613211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038996+0000","last_change":"2024-04-15T15:22:33.301064+0000","last_active":"2024-04-15T15:22:35.038996+0000","last_peered":"2024-04-15T15:22:35.038996+0000","last_clean":"2024-04-15T15:22:35.038996+0000","last_became_active":"2024-04-15T15:22:33.299748+0000","last_became_peered":"2024-04-15T15:22:33.299748+0000","last_unstale":"2024-04-15T15:22:35.038996+0000","last_undegraded":"2024-04-15T15:22:35.038996+0000","last_fullsized":"2024-04-15T15:22:35.038996+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:14:54.794343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705249+0000","last_change":"2024-04-15T15:22:33.314561+0000","last_active":"2024-04-15T15:22:34.705249+0000","last_peered":"2024-04-15T15:22:34.705249+0000","last_clean":"2024-04-15T15:22:34.705249+0000","last_became_active":"2024-04-15T15:22:33.314040+0000","last_became_peered":"2024-04-15T15:22:33.314040+0000","last_unstale":"2024-04-15T15:22:34.705249+0000","last_undegraded":"2024-04-15T15:22:34.705249+0000","last_fullsized":"2024-04-15T15:22:34.705249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:26.586364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038894+0000","last_change":"2024-04-15T15:22:32.295369+0000","last_active":"2024-04-15T15:22:35.038894+0000","last_peered":"2024-04-15T15:22:35.038894+0000","last_clean":"2024-04-15T15:22:35.038894+0000","last_became_active":"2024-04-15T15:22:32.295054+0000","last_became_peered":"2024-04-15T15:22:32.295054+0000","last_unstale":"2024-04-15T15:22:35.038894+0000","last_undegraded":"2024-04-15T15:22:35.038894+0000","last_fullsized":"2024-04-15T15:22:35.038894+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039729+0000","last_change":"2024-04-15T15:22:33.311968+0000","last_active":"2024-04-15T15:22:35.039729+0000","last_peered":"2024-04-15T15:22:35.039729+0000","last_clean":"2024-04-15T15:22:35.039729+0000","last_became_active":"2024-04-15T15:22:33.311690+0000","last_became_peered":"2024-04-15T15:22:33.311690+0000","last_unstale":"2024-04-15T15:22:35.039729+0000","last_undegraded":"2024-04-15T15:22:35.039729+0000","last_fullsized":"2024-04-15T15:22:35.039729+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:49:13.345861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565472+0000","last_change":"2024-04-15T15:22:32.298663+0000","last_active":"2024-04-15T15:22:34.565472+0000","last_peered":"2024-04-15T15:22:34.565472+0000","last_clean":"2024-04-15T15:22:34.565472+0000","last_became_active":"2024-04-15T15:22:32.298296+0000","last_became_peered":"2024-04-15T15:22:32.298296+0000","last_unstale":"2024-04-15T15:22:34.565472+0000","last_undegraded":"2024-04-15T15:22:34.565472+0000","last_fullsized":"2024-04-15T15:22:34.565472+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538568+0000","last_change":"2024-04-15T15:22:33.296800+0000","last_active":"2024-04-15T15:22:34.538568+0000","last_peered":"2024-04-15T15:22:34.538568+0000","last_clean":"2024-04-15T15:22:34.538568+0000","last_became_active":"2024-04-15T15:22:33.296491+0000","last_became_peered":"2024-04-15T15:22:33.296491+0000","last_unstale":"2024-04-15T15:22:34.538568+0000","last_undegraded":"2024-04-15T15:22:34.538568+0000","last_fullsized":"2024-04-15T15:22:34.538568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:30:58.162485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039847+0000","last_change":"2024-04-15T15:22:32.301621+0000","last_active":"2024-04-15T15:22:35.039847+0000","last_peered":"2024-04-15T15:22:35.039847+0000","last_clean":"2024-04-15T15:22:35.039847+0000","last_became_active":"2024-04-15T15:22:32.301253+0000","last_became_peered":"2024-04-15T15:22:32.301253+0000","last_unstale":"2024-04-15T15:22:35.039847+0000","last_undegraded":"2024-04-15T15:22:35.039847+0000","last_fullsized":"2024-04-15T15:22:35.039847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295542+0000","last_change":"2024-04-15T15:22:33.296469+0000","last_active":"2024-04-15T15:22:34.295542+0000","last_peered":"2024-04-15T15:22:34.295542+0000","last_clean":"2024-04-15T15:22:34.295542+0000","last_became_active":"2024-04-15T15:22:33.296158+0000","last_became_peered":"2024-04-15T15:22:33.296158+0000","last_unstale":"2024-04-15T15:22:34.295542+0000","last_undegraded":"2024-04-15T15:22:34.295542+0000","last_fullsized":"2024-04-15T15:22:34.295542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:51:51.191240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538325+0000","last_change":"2024-04-15T15:22:32.297657+0000","last_active":"2024-04-15T15:22:34.538325+0000","last_peered":"2024-04-15T15:22:34.538325+0000","last_clean":"2024-04-15T15:22:34.538325+0000","last_became_active":"2024-04-15T15:22:32.297419+0000","last_became_peered":"2024-04-15T15:22:32.297419+0000","last_unstale":"2024-04-15T15:22:34.538325+0000","last_undegraded":"2024-04-15T15:22:34.538325+0000","last_fullsized":"2024-04-15T15:22:34.538325+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565200+0000","last_change":"2024-04-15T15:22:33.300618+0000","last_active":"2024-04-15T15:22:34.565200+0000","last_peered":"2024-04-15T15:22:34.565200+0000","last_clean":"2024-04-15T15:22:34.565200+0000","last_became_active":"2024-04-15T15:22:33.300356+0000","last_became_peered":"2024-04-15T15:22:33.300356+0000","last_unstale":"2024-04-15T15:22:34.565200+0000","last_undegraded":"2024-04-15T15:22:34.565200+0000","last_fullsized":"2024-04-15T15:22:34.565200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:25:09.987700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305834+0000","last_change":"2024-04-15T15:22:32.293808+0000","last_active":"2024-04-15T15:22:34.305834+0000","last_peered":"2024-04-15T15:22:34.305834+0000","last_clean":"2024-04-15T15:22:34.305834+0000","last_became_active":"2024-04-15T15:22:32.293474+0000","last_became_peered":"2024-04-15T15:22:32.293474+0000","last_unstale":"2024-04-15T15:22:34.305834+0000","last_undegraded":"2024-04-15T15:22:34.305834+0000","last_fullsized":"2024-04-15T15:22:34.305834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:02:14.685440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305640+0000","last_change":"2024-04-15T15:22:33.305485+0000","last_active":"2024-04-15T15:22:34.305640+0000","last_peered":"2024-04-15T15:22:34.305640+0000","last_clean":"2024-04-15T15:22:34.305640+0000","last_became_active":"2024-04-15T15:22:33.305123+0000","last_became_peered":"2024-04-15T15:22:33.305123+0000","last_unstale":"2024-04-15T15:22:34.305640+0000","last_undegraded":"2024-04-15T15:22:34.305640+0000","last_fullsized":"2024-04-15T15:22:34.305640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:48.797544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705020+0000","last_change":"2024-04-15T15:22:32.299916+0000","last_active":"2024-04-15T15:22:34.705020+0000","last_peered":"2024-04-15T15:22:34.705020+0000","last_clean":"2024-04-15T15:22:34.705020+0000","last_became_active":"2024-04-15T15:22:32.299576+0000","last_became_peered":"2024-04-15T15:22:32.299576+0000","last_unstale":"2024-04-15T15:22:34.705020+0000","last_undegraded":"2024-04-15T15:22:34.705020+0000","last_fullsized":"2024-04-15T15:22:34.705020+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538525+0000","last_change":"2024-04-15T15:22:33.296037+0000","last_active":"2024-04-15T15:22:34.538525+0000","last_peered":"2024-04-15T15:22:34.538525+0000","last_clean":"2024-04-15T15:22:34.538525+0000","last_became_active":"2024-04-15T15:22:33.295570+0000","last_became_peered":"2024-04-15T15:22:33.295570+0000","last_unstale":"2024-04-15T15:22:34.538525+0000","last_undegraded":"2024-04-15T15:22:34.538525+0000","last_fullsized":"2024-04-15T15:22:34.538525+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:42:06.582334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039045+0000","last_change":"2024-04-15T15:22:32.301232+0000","last_active":"2024-04-15T15:22:35.039045+0000","last_peered":"2024-04-15T15:22:35.039045+0000","last_clean":"2024-04-15T15:22:35.039045+0000","last_became_active":"2024-04-15T15:22:32.300885+0000","last_became_peered":"2024-04-15T15:22:32.300885+0000","last_unstale":"2024-04-15T15:22:35.039045+0000","last_undegraded":"2024-04-15T15:22:35.039045+0000","last_fullsized":"2024-04-15T15:22:35.039045+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295257+0000","last_change":"2024-04-15T15:22:33.305865+0000","last_active":"2024-04-15T15:22:34.295257+0000","last_peered":"2024-04-15T15:22:34.295257+0000","last_clean":"2024-04-15T15:22:34.295257+0000","last_became_active":"2024-04-15T15:22:33.305605+0000","last_became_peered":"2024-04-15T15:22:33.305605+0000","last_unstale":"2024-04-15T15:22:34.295257+0000","last_undegraded":"2024-04-15T15:22:34.295257+0000","last_fullsized":"2024-04-15T15:22:34.295257+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:17:04.067763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.537966+0000","last_change":"2024-04-15T15:22:32.288888+0000","last_active":"2024-04-15T15:22:34.537966+0000","last_peered":"2024-04-15T15:22:34.537966+0000","last_clean":"2024-04-15T15:22:34.537966+0000","last_became_active":"2024-04-15T15:22:32.288559+0000","last_became_peered":"2024-04-15T15:22:32.288559+0000","last_unstale":"2024-04-15T15:22:34.537966+0000","last_undegraded":"2024-04-15T15:22:34.537966+0000","last_fullsized":"2024-04-15T15:22:34.537966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039716+0000","last_change":"2024-04-15T15:22:33.296928+0000","last_active":"2024-04-15T15:22:35.039716+0000","last_peered":"2024-04-15T15:22:35.039716+0000","last_clean":"2024-04-15T15:22:35.039716+0000","last_became_active":"2024-04-15T15:22:33.296618+0000","last_became_peered":"2024-04-15T15:22:33.296618+0000","last_unstale":"2024-04-15T15:22:35.039716+0000","last_undegraded":"2024-04-15T15:22:35.039716+0000","last_fullsized":"2024-04-15T15:22:35.039716+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:46.712200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295636+0000","last_change":"2024-04-15T15:22:32.292353+0000","last_active":"2024-04-15T15:22:34.295636+0000","last_peered":"2024-04-15T15:22:34.295636+0000","last_clean":"2024-04-15T15:22:34.295636+0000","last_became_active":"2024-04-15T15:22:32.292043+0000","last_became_peered":"2024-04-15T15:22:32.292043+0000","last_unstale":"2024-04-15T15:22:34.295636+0000","last_undegraded":"2024-04-15T15:22:34.295636+0000","last_fullsized":"2024-04-15T15:22:34.295636+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:18:34.871504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705618+0000","last_change":"2024-04-15T15:22:33.309220+0000","last_active":"2024-04-15T15:22:34.705618+0000","last_peered":"2024-04-15T15:22:34.705618+0000","last_clean":"2024-04-15T15:22:34.705618+0000","last_became_active":"2024-04-15T15:22:33.308201+0000","last_became_peered":"2024-04-15T15:22:33.308201+0000","last_unstale":"2024-04-15T15:22:34.705618+0000","last_undegraded":"2024-04-15T15:22:34.705618+0000","last_fullsized":"2024-04-15T15:22:34.705618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:18.339496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306092+0000","last_change":"2024-04-15T15:22:32.294706+0000","last_active":"2024-04-15T15:22:34.306092+0000","last_peered":"2024-04-15T15:22:34.306092+0000","last_clean":"2024-04-15T15:22:34.306092+0000","last_became_active":"2024-04-15T15:22:32.294353+0000","last_became_peered":"2024-04-15T15:22:32.294353+0000","last_unstale":"2024-04-15T15:22:34.306092+0000","last_undegraded":"2024-04-15T15:22:34.306092+0000","last_fullsized":"2024-04-15T15:22:34.306092+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:53:38.512384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565590+0000","last_change":"2024-04-15T15:22:33.309489+0000","last_active":"2024-04-15T15:22:34.565590+0000","last_peered":"2024-04-15T15:22:34.565590+0000","last_clean":"2024-04-15T15:22:34.565590+0000","last_became_active":"2024-04-15T15:22:33.309241+0000","last_became_peered":"2024-04-15T15:22:33.309241+0000","last_unstale":"2024-04-15T15:22:34.565590+0000","last_undegraded":"2024-04-15T15:22:34.565590+0000","last_fullsized":"2024-04-15T15:22:34.565590+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:11:33.617468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306285+0000","last_change":"2024-04-15T15:22:32.294321+0000","last_active":"2024-04-15T15:22:34.306285+0000","last_peered":"2024-04-15T15:22:34.306285+0000","last_clean":"2024-04-15T15:22:34.306285+0000","last_became_active":"2024-04-15T15:22:32.294043+0000","last_became_peered":"2024-04-15T15:22:32.294043+0000","last_unstale":"2024-04-15T15:22:34.306285+0000","last_undegraded":"2024-04-15T15:22:34.306285+0000","last_fullsized":"2024-04-15T15:22:34.306285+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:47.063192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538601+0000","last_change":"2024-04-15T15:22:33.302287+0000","last_active":"2024-04-15T15:22:34.538601+0000","last_peered":"2024-04-15T15:22:34.538601+0000","last_clean":"2024-04-15T15:22:34.538601+0000","last_became_active":"2024-04-15T15:22:33.302020+0000","last_became_peered":"2024-04-15T15:22:33.302020+0000","last_unstale":"2024-04-15T15:22:34.538601+0000","last_undegraded":"2024-04-15T15:22:34.538601+0000","last_fullsized":"2024-04-15T15:22:34.538601+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:49:13.345861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295609+0000","last_change":"2024-04-15T15:22:32.298894+0000","last_active":"2024-04-15T15:22:34.295609+0000","last_peered":"2024-04-15T15:22:34.295609+0000","last_clean":"2024-04-15T15:22:34.295609+0000","last_became_active":"2024-04-15T15:22:32.298534+0000","last_became_peered":"2024-04-15T15:22:32.298534+0000","last_unstale":"2024-04-15T15:22:34.295609+0000","last_undegraded":"2024-04-15T15:22:34.295609+0000","last_fullsized":"2024-04-15T15:22:34.295609+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:17:28.183508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565266+0000","last_change":"2024-04-15T15:22:33.308028+0000","last_active":"2024-04-15T15:22:34.565266+0000","last_peered":"2024-04-15T15:22:34.565266+0000","last_clean":"2024-04-15T15:22:34.565266+0000","last_became_active":"2024-04-15T15:22:33.307745+0000","last_became_peered":"2024-04-15T15:22:33.307745+0000","last_unstale":"2024-04-15T15:22:34.565266+0000","last_undegraded":"2024-04-15T15:22:34.565266+0000","last_fullsized":"2024-04-15T15:22:34.565266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:42:06.582334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295608+0000","last_change":"2024-04-15T15:22:32.287177+0000","last_active":"2024-04-15T15:22:34.295608+0000","last_peered":"2024-04-15T15:22:34.295608+0000","last_clean":"2024-04-15T15:22:34.295608+0000","last_became_active":"2024-04-15T15:22:32.286887+0000","last_became_peered":"2024-04-15T15:22:32.286887+0000","last_unstale":"2024-04-15T15:22:34.295608+0000","last_undegraded":"2024-04-15T15:22:34.295608+0000","last_fullsized":"2024-04-15T15:22:34.295608+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:02:14.685440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295451+0000","last_change":"2024-04-15T15:22:33.294006+0000","last_active":"2024-04-15T15:22:34.295451+0000","last_peered":"2024-04-15T15:22:34.295451+0000","last_clean":"2024-04-15T15:22:34.295451+0000","last_became_active":"2024-04-15T15:22:33.293049+0000","last_became_peered":"2024-04-15T15:22:33.293049+0000","last_unstale":"2024-04-15T15:22:34.295451+0000","last_undegraded":"2024-04-15T15:22:34.295451+0000","last_fullsized":"2024-04-15T15:22:34.295451+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:48.797544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039203+0000","last_change":"2024-04-15T15:22:32.301053+0000","last_active":"2024-04-15T15:22:35.039203+0000","last_peered":"2024-04-15T15:22:35.039203+0000","last_clean":"2024-04-15T15:22:35.039203+0000","last_became_active":"2024-04-15T15:22:32.300755+0000","last_became_peered":"2024-04-15T15:22:32.300755+0000","last_unstale":"2024-04-15T15:22:35.039203+0000","last_undegraded":"2024-04-15T15:22:35.039203+0000","last_fullsized":"2024-04-15T15:22:35.039203+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705129+0000","last_change":"2024-04-15T15:22:33.307952+0000","last_active":"2024-04-15T15:22:34.705129+0000","last_peered":"2024-04-15T15:22:34.705129+0000","last_clean":"2024-04-15T15:22:34.705129+0000","last_became_active":"2024-04-15T15:22:33.307634+0000","last_became_peered":"2024-04-15T15:22:33.307634+0000","last_unstale":"2024-04-15T15:22:34.705129+0000","last_undegraded":"2024-04-15T15:22:34.705129+0000","last_fullsized":"2024-04-15T15:22:34.705129+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:11:33.617468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038515+0000","last_change":"2024-04-15T15:22:32.293921+0000","last_active":"2024-04-15T15:22:35.038515+0000","last_peered":"2024-04-15T15:22:35.038515+0000","last_clean":"2024-04-15T15:22:35.038515+0000","last_became_active":"2024-04-15T15:22:32.293641+0000","last_became_peered":"2024-04-15T15:22:32.293641+0000","last_unstale":"2024-04-15T15:22:35.038515+0000","last_undegraded":"2024-04-15T15:22:35.038515+0000","last_fullsized":"2024-04-15T15:22:35.038515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:09:33.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295341+0000","last_change":"2024-04-15T15:22:33.292750+0000","last_active":"2024-04-15T15:22:34.295341+0000","last_peered":"2024-04-15T15:22:34.295341+0000","last_clean":"2024-04-15T15:22:34.295341+0000","last_became_active":"2024-04-15T15:22:33.292441+0000","last_became_peered":"2024-04-15T15:22:33.292441+0000","last_unstale":"2024-04-15T15:22:34.295341+0000","last_undegraded":"2024-04-15T15:22:34.295341+0000","last_fullsized":"2024-04-15T15:22:34.295341+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:26.586364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705012+0000","last_change":"2024-04-15T15:22:32.299339+0000","last_active":"2024-04-15T15:22:34.705012+0000","last_peered":"2024-04-15T15:22:34.705012+0000","last_clean":"2024-04-15T15:22:34.705012+0000","last_became_active":"2024-04-15T15:22:32.298921+0000","last_became_peered":"2024-04-15T15:22:32.298921+0000","last_unstale":"2024-04-15T15:22:34.705012+0000","last_undegraded":"2024-04-15T15:22:34.705012+0000","last_fullsized":"2024-04-15T15:22:34.705012+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:21:12.998104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295625+0000","last_change":"2024-04-15T15:22:33.296963+0000","last_active":"2024-04-15T15:22:34.295625+0000","last_peered":"2024-04-15T15:22:34.295625+0000","last_clean":"2024-04-15T15:22:34.295625+0000","last_became_active":"2024-04-15T15:22:33.296639+0000","last_became_peered":"2024-04-15T15:22:33.296639+0000","last_unstale":"2024-04-15T15:22:34.295625+0000","last_undegraded":"2024-04-15T15:22:34.295625+0000","last_fullsized":"2024-04-15T15:22:34.295625+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:26:40.045991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306207+0000","last_change":"2024-04-15T15:22:32.299157+0000","last_active":"2024-04-15T15:22:34.306207+0000","last_peered":"2024-04-15T15:22:34.306207+0000","last_clean":"2024-04-15T15:22:34.306207+0000","last_became_active":"2024-04-15T15:22:32.298836+0000","last_became_peered":"2024-04-15T15:22:32.298836+0000","last_unstale":"2024-04-15T15:22:34.306207+0000","last_undegraded":"2024-04-15T15:22:34.306207+0000","last_fullsized":"2024-04-15T15:22:34.306207+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:17:28.183508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306413+0000","last_change":"2024-04-15T15:22:33.305470+0000","last_active":"2024-04-15T15:22:34.306413+0000","last_peered":"2024-04-15T15:22:34.306413+0000","last_clean":"2024-04-15T15:22:34.306413+0000","last_became_active":"2024-04-15T15:22:33.305182+0000","last_became_peered":"2024-04-15T15:22:33.305182+0000","last_unstale":"2024-04-15T15:22:34.306413+0000","last_undegraded":"2024-04-15T15:22:34.306413+0000","last_fullsized":"2024-04-15T15:22:34.306413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:40:22.473471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538605+0000","last_change":"2024-04-15T15:22:32.301976+0000","last_active":"2024-04-15T15:22:34.538605+0000","last_peered":"2024-04-15T15:22:34.538605+0000","last_clean":"2024-04-15T15:22:34.538605+0000","last_became_active":"2024-04-15T15:22:32.301692+0000","last_became_peered":"2024-04-15T15:22:32.301692+0000","last_unstale":"2024-04-15T15:22:34.538605+0000","last_undegraded":"2024-04-15T15:22:34.538605+0000","last_fullsized":"2024-04-15T15:22:34.538605+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538486+0000","last_change":"2024-04-15T15:22:33.304545+0000","last_active":"2024-04-15T15:22:34.538486+0000","last_peered":"2024-04-15T15:22:34.538486+0000","last_clean":"2024-04-15T15:22:34.538486+0000","last_became_active":"2024-04-15T15:22:33.304196+0000","last_became_peered":"2024-04-15T15:22:33.304196+0000","last_unstale":"2024-04-15T15:22:34.538486+0000","last_undegraded":"2024-04-15T15:22:34.538486+0000","last_fullsized":"2024-04-15T15:22:34.538486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:25:09.987700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039681+0000","last_change":"2024-04-15T15:22:32.299415+0000","last_active":"2024-04-15T15:22:35.039681+0000","last_peered":"2024-04-15T15:22:35.039681+0000","last_clean":"2024-04-15T15:22:35.039681+0000","last_became_active":"2024-04-15T15:22:32.299078+0000","last_became_peered":"2024-04-15T15:22:32.299078+0000","last_unstale":"2024-04-15T15:22:35.039681+0000","last_undegraded":"2024-04-15T15:22:35.039681+0000","last_fullsized":"2024-04-15T15:22:35.039681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:05:16.011118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039501+0000","last_change":"2024-04-15T15:22:33.291429+0000","last_active":"2024-04-15T15:22:35.039501+0000","last_peered":"2024-04-15T15:22:35.039501+0000","last_clean":"2024-04-15T15:22:35.039501+0000","last_became_active":"2024-04-15T15:22:33.291151+0000","last_became_peered":"2024-04-15T15:22:33.291151+0000","last_unstale":"2024-04-15T15:22:35.039501+0000","last_undegraded":"2024-04-15T15:22:35.039501+0000","last_fullsized":"2024-04-15T15:22:35.039501+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.554681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039454+0000","last_change":"2024-04-15T15:22:32.301462+0000","last_active":"2024-04-15T15:22:35.039454+0000","last_peered":"2024-04-15T15:22:35.039454+0000","last_clean":"2024-04-15T15:22:35.039454+0000","last_became_active":"2024-04-15T15:22:32.301176+0000","last_became_peered":"2024-04-15T15:22:32.301176+0000","last_unstale":"2024-04-15T15:22:35.039454+0000","last_undegraded":"2024-04-15T15:22:35.039454+0000","last_fullsized":"2024-04-15T15:22:35.039454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705360+0000","last_change":"2024-04-15T15:22:33.309028+0000","last_active":"2024-04-15T15:22:34.705360+0000","last_peered":"2024-04-15T15:22:34.705360+0000","last_clean":"2024-04-15T15:22:34.705360+0000","last_became_active":"2024-04-15T15:22:33.308424+0000","last_became_peered":"2024-04-15T15:22:33.308424+0000","last_unstale":"2024-04-15T15:22:34.705360+0000","last_undegraded":"2024-04-15T15:22:34.705360+0000","last_fullsized":"2024-04-15T15:22:34.705360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:33.588528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039242+0000","last_change":"2024-04-15T15:22:32.290430+0000","last_active":"2024-04-15T15:22:35.039242+0000","last_peered":"2024-04-15T15:22:35.039242+0000","last_clean":"2024-04-15T15:22:35.039242+0000","last_became_active":"2024-04-15T15:22:32.290116+0000","last_became_peered":"2024-04-15T15:22:32.290116+0000","last_unstale":"2024-04-15T15:22:35.039242+0000","last_undegraded":"2024-04-15T15:22:35.039242+0000","last_fullsized":"2024-04-15T15:22:35.039242+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:43:31.231190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565305+0000","last_change":"2024-04-15T15:22:33.303094+0000","last_active":"2024-04-15T15:22:34.565305+0000","last_peered":"2024-04-15T15:22:34.565305+0000","last_clean":"2024-04-15T15:22:34.565305+0000","last_became_active":"2024-04-15T15:22:33.302824+0000","last_became_peered":"2024-04-15T15:22:33.302824+0000","last_unstale":"2024-04-15T15:22:34.565305+0000","last_undegraded":"2024-04-15T15:22:34.565305+0000","last_fullsized":"2024-04-15T15:22:34.565305+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:22:11.084708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039278+0000","last_change":"2024-04-15T15:22:32.299094+0000","last_active":"2024-04-15T15:22:35.039278+0000","last_peered":"2024-04-15T15:22:35.039278+0000","last_clean":"2024-04-15T15:22:35.039278+0000","last_became_active":"2024-04-15T15:22:32.298792+0000","last_became_peered":"2024-04-15T15:22:32.298792+0000","last_unstale":"2024-04-15T15:22:35.039278+0000","last_undegraded":"2024-04-15T15:22:35.039278+0000","last_fullsized":"2024-04-15T15:22:35.039278+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:10:15.989931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305979+0000","last_change":"2024-04-15T15:22:33.296445+0000","last_active":"2024-04-15T15:22:34.305979+0000","last_peered":"2024-04-15T15:22:34.305979+0000","last_clean":"2024-04-15T15:22:34.305979+0000","last_became_active":"2024-04-15T15:22:33.296187+0000","last_became_peered":"2024-04-15T15:22:33.296187+0000","last_unstale":"2024-04-15T15:22:34.305979+0000","last_undegraded":"2024-04-15T15:22:34.305979+0000","last_fullsized":"2024-04-15T15:22:34.305979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:46:17.698031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705414+0000","last_change":"2024-04-15T15:22:32.298790+0000","last_active":"2024-04-15T15:22:34.705414+0000","last_peered":"2024-04-15T15:22:34.705414+0000","last_clean":"2024-04-15T15:22:34.705414+0000","last_became_active":"2024-04-15T15:22:32.298475+0000","last_became_peered":"2024-04-15T15:22:32.298475+0000","last_unstale":"2024-04-15T15:22:34.705414+0000","last_undegraded":"2024-04-15T15:22:34.705414+0000","last_fullsized":"2024-04-15T15:22:34.705414+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:45.706167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295502+0000","last_change":"2024-04-15T15:22:33.305871+0000","last_active":"2024-04-15T15:22:34.295502+0000","last_peered":"2024-04-15T15:22:34.295502+0000","last_clean":"2024-04-15T15:22:34.295502+0000","last_became_active":"2024-04-15T15:22:33.305213+0000","last_became_peered":"2024-04-15T15:22:33.305213+0000","last_unstale":"2024-04-15T15:22:34.295502+0000","last_undegraded":"2024-04-15T15:22:34.295502+0000","last_fullsized":"2024-04-15T15:22:34.295502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:18.339496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039592+0000","last_change":"2024-04-15T15:22:32.293012+0000","last_active":"2024-04-15T15:22:35.039592+0000","last_peered":"2024-04-15T15:22:35.039592+0000","last_clean":"2024-04-15T15:22:35.039592+0000","last_became_active":"2024-04-15T15:22:32.292728+0000","last_became_peered":"2024-04-15T15:22:32.292728+0000","last_unstale":"2024-04-15T15:22:35.039592+0000","last_undegraded":"2024-04-15T15:22:35.039592+0000","last_fullsized":"2024-04-15T15:22:35.039592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:17:38.233283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039522+0000","last_change":"2024-04-15T15:22:33.309264+0000","last_active":"2024-04-15T15:22:35.039522+0000","last_peered":"2024-04-15T15:22:35.039522+0000","last_clean":"2024-04-15T15:22:35.039522+0000","last_became_active":"2024-04-15T15:22:33.309005+0000","last_became_peered":"2024-04-15T15:22:33.309005+0000","last_unstale":"2024-04-15T15:22:35.039522+0000","last_undegraded":"2024-04-15T15:22:35.039522+0000","last_fullsized":"2024-04-15T15:22:35.039522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:42:06.582334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306292+0000","last_change":"2024-04-15T15:22:32.295146+0000","last_active":"2024-04-15T15:22:34.306292+0000","last_peered":"2024-04-15T15:22:34.306292+0000","last_clean":"2024-04-15T15:22:34.306292+0000","last_became_active":"2024-04-15T15:22:32.294842+0000","last_became_peered":"2024-04-15T15:22:32.294842+0000","last_unstale":"2024-04-15T15:22:34.306292+0000","last_undegraded":"2024-04-15T15:22:34.306292+0000","last_fullsized":"2024-04-15T15:22:34.306292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:18:34.871504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705452+0000","last_change":"2024-04-15T15:22:33.309039+0000","last_active":"2024-04-15T15:22:34.705452+0000","last_peered":"2024-04-15T15:22:34.705452+0000","last_clean":"2024-04-15T15:22:34.705452+0000","last_became_active":"2024-04-15T15:22:33.308548+0000","last_became_peered":"2024-04-15T15:22:33.308548+0000","last_unstale":"2024-04-15T15:22:34.705452+0000","last_undegraded":"2024-04-15T15:22:34.705452+0000","last_fullsized":"2024-04-15T15:22:34.705452+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:48.797544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039339+0000","last_change":"2024-04-15T15:22:32.295490+0000","last_active":"2024-04-15T15:22:35.039339+0000","last_peered":"2024-04-15T15:22:35.039339+0000","last_clean":"2024-04-15T15:22:35.039339+0000","last_became_active":"2024-04-15T15:22:32.295129+0000","last_became_peered":"2024-04-15T15:22:32.295129+0000","last_unstale":"2024-04-15T15:22:35.039339+0000","last_undegraded":"2024-04-15T15:22:35.039339+0000","last_fullsized":"2024-04-15T15:22:35.039339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:02:52.088900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295379+0000","last_change":"2024-04-15T15:22:33.303705+0000","last_active":"2024-04-15T15:22:34.295379+0000","last_peered":"2024-04-15T15:22:34.295379+0000","last_clean":"2024-04-15T15:22:34.295379+0000","last_became_active":"2024-04-15T15:22:33.303410+0000","last_became_peered":"2024-04-15T15:22:33.303410+0000","last_unstale":"2024-04-15T15:22:34.295379+0000","last_undegraded":"2024-04-15T15:22:34.295379+0000","last_fullsized":"2024-04-15T15:22:34.295379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:33.588528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565454+0000","last_change":"2024-04-15T15:22:32.294246+0000","last_active":"2024-04-15T15:22:34.565454+0000","last_peered":"2024-04-15T15:22:34.565454+0000","last_clean":"2024-04-15T15:22:34.565454+0000","last_became_active":"2024-04-15T15:22:32.293952+0000","last_became_peered":"2024-04-15T15:22:32.293952+0000","last_unstale":"2024-04-15T15:22:34.565454+0000","last_undegraded":"2024-04-15T15:22:34.565454+0000","last_fullsized":"2024-04-15T15:22:34.565454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565358+0000","last_change":"2024-04-15T15:22:33.294263+0000","last_active":"2024-04-15T15:22:34.565358+0000","last_peered":"2024-04-15T15:22:34.565358+0000","last_clean":"2024-04-15T15:22:34.565358+0000","last_became_active":"2024-04-15T15:22:33.293971+0000","last_became_peered":"2024-04-15T15:22:33.293971+0000","last_unstale":"2024-04-15T15:22:34.565358+0000","last_undegraded":"2024-04-15T15:22:34.565358+0000","last_fullsized":"2024-04-15T15:22:34.565358+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:49:13.345861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565507+0000","last_change":"2024-04-15T15:22:32.288288+0000","last_active":"2024-04-15T15:22:34.565507+0000","last_peered":"2024-04-15T15:22:34.565507+0000","last_clean":"2024-04-15T15:22:34.565507+0000","last_became_active":"2024-04-15T15:22:32.287969+0000","last_became_peered":"2024-04-15T15:22:32.287969+0000","last_unstale":"2024-04-15T15:22:34.565507+0000","last_undegraded":"2024-04-15T15:22:34.565507+0000","last_fullsized":"2024-04-15T15:22:34.565507+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039057+0000","last_change":"2024-04-15T15:22:33.300140+0000","last_active":"2024-04-15T15:22:35.039057+0000","last_peered":"2024-04-15T15:22:35.039057+0000","last_clean":"2024-04-15T15:22:35.039057+0000","last_became_active":"2024-04-15T15:22:33.299715+0000","last_became_peered":"2024-04-15T15:22:33.299715+0000","last_unstale":"2024-04-15T15:22:35.039057+0000","last_undegraded":"2024-04-15T15:22:35.039057+0000","last_fullsized":"2024-04-15T15:22:35.039057+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:18:22.291418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038645+0000","last_change":"2024-04-15T15:22:32.292570+0000","last_active":"2024-04-15T15:22:35.038645+0000","last_peered":"2024-04-15T15:22:35.038645+0000","last_clean":"2024-04-15T15:22:35.038645+0000","last_became_active":"2024-04-15T15:22:32.292275+0000","last_became_peered":"2024-04-15T15:22:32.292275+0000","last_unstale":"2024-04-15T15:22:35.038645+0000","last_undegraded":"2024-04-15T15:22:35.038645+0000","last_fullsized":"2024-04-15T15:22:35.038645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:01.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538124+0000","last_change":"2024-04-15T15:22:33.295370+0000","last_active":"2024-04-15T15:22:34.538124+0000","last_peered":"2024-04-15T15:22:34.538124+0000","last_clean":"2024-04-15T15:22:34.538124+0000","last_became_active":"2024-04-15T15:22:33.295068+0000","last_became_peered":"2024-04-15T15:22:33.295068+0000","last_unstale":"2024-04-15T15:22:34.538124+0000","last_undegraded":"2024-04-15T15:22:34.538124+0000","last_fullsized":"2024-04-15T15:22:34.538124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:14:54.794343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.537995+0000","last_change":"2024-04-15T15:22:32.297615+0000","last_active":"2024-04-15T15:22:34.537995+0000","last_peered":"2024-04-15T15:22:34.537995+0000","last_clean":"2024-04-15T15:22:34.537995+0000","last_became_active":"2024-04-15T15:22:32.297385+0000","last_became_peered":"2024-04-15T15:22:32.297385+0000","last_unstale":"2024-04-15T15:22:34.537995+0000","last_undegraded":"2024-04-15T15:22:34.537995+0000","last_fullsized":"2024-04-15T15:22:34.537995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705270+0000","last_change":"2024-04-15T15:22:33.307911+0000","last_active":"2024-04-15T15:22:34.705270+0000","last_peered":"2024-04-15T15:22:34.705270+0000","last_clean":"2024-04-15T15:22:34.705270+0000","last_became_active":"2024-04-15T15:22:33.307610+0000","last_became_peered":"2024-04-15T15:22:33.307610+0000","last_unstale":"2024-04-15T15:22:34.705270+0000","last_undegraded":"2024-04-15T15:22:34.705270+0000","last_fullsized":"2024-04-15T15:22:34.705270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:17:04.067763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295120+0000","last_change":"2024-04-15T15:22:32.292214+0000","last_active":"2024-04-15T15:22:34.295120+0000","last_peered":"2024-04-15T15:22:34.295120+0000","last_clean":"2024-04-15T15:22:34.295120+0000","last_became_active":"2024-04-15T15:22:32.291892+0000","last_became_peered":"2024-04-15T15:22:32.291892+0000","last_unstale":"2024-04-15T15:22:34.295120+0000","last_undegraded":"2024-04-15T15:22:34.295120+0000","last_fullsized":"2024-04-15T15:22:34.295120+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:47.063192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306093+0000","last_change":"2024-04-15T15:22:33.301528+0000","last_active":"2024-04-15T15:22:34.306093+0000","last_peered":"2024-04-15T15:22:34.306093+0000","last_clean":"2024-04-15T15:22:34.306093+0000","last_became_active":"2024-04-15T15:22:33.298894+0000","last_became_peered":"2024-04-15T15:22:33.298894+0000","last_unstale":"2024-04-15T15:22:34.306093+0000","last_undegraded":"2024-04-15T15:22:34.306093+0000","last_fullsized":"2024-04-15T15:22:34.306093+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:23:51.004333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705252+0000","last_change":"2024-04-15T15:22:32.299060+0000","last_active":"2024-04-15T15:22:34.705252+0000","last_peered":"2024-04-15T15:22:34.705252+0000","last_clean":"2024-04-15T15:22:34.705252+0000","last_became_active":"2024-04-15T15:22:32.298599+0000","last_became_peered":"2024-04-15T15:22:32.298599+0000","last_unstale":"2024-04-15T15:22:34.705252+0000","last_undegraded":"2024-04-15T15:22:34.705252+0000","last_fullsized":"2024-04-15T15:22:34.705252+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:47.063192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705488+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:22:34.705488+0000","last_peered":"2024-04-15T15:22:34.705488+0000","last_clean":"2024-04-15T15:22:34.705488+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:22:34.705488+0000","last_undegraded":"2024-04-15T15:22:34.705488+0000","last_fullsized":"2024-04-15T15:22:34.705488+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":1060864,"data_stored":787081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":195613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":458752,"data_stored":195143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":458752,"data_stored":195635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":196677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":195101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":195121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":785433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:22:41.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:22:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.905+0000 7fd08ec88640 1 Processor -- start 2024-04-15T15:22:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.905+0000 7fd08ec88640 1 -- start start 2024-04-15T15:22:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.905+0000 7fd08ec88640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd08815e700 0x7fd0880b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.905+0000 7fd08ec88640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.905+0000 7fd08ec88640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0880b47a0 con 0x7fd08815eae0 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.905+0000 7fd08ec88640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd0880b53b0 con 0x7fd08815e700 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.905+0000 7fd08ec88640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd0880b4a80 con 0x7fd0880d2360 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd087fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd08ca24640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd08815e700 0x7fd0880b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd08d225640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fd08815eae0 0x7fd0880cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:45508/0 (socket says 172.21.15.12:45508) 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd08d225640 1 -- 172.21.15.12:0/2722675085 learned_addr learned my addr 172.21.15.12:0/2722675085 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd08ca24640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd08815e700 0x7fd0880b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:40732/0 (socket says 172.21.15.12:40732) 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd087fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:42848/0 (socket says 172.21.15.12:42848) 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd087fff640 1 -- 172.21.15.12:0/2722675085 >> v2:172.21.15.55:3300/0 conn(0x7fd08815e700 msgr2=0x7fd0880b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd087fff640 1 --2- 172.21.15.12:0/2722675085 >> v2:172.21.15.55:3300/0 conn(0x7fd08815e700 0x7fd0880b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd087fff640 1 -- 172.21.15.12:0/2722675085 >> v1:172.21.15.55:6789/0 conn(0x7fd08815eae0 legacy=0x7fd0880cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd0877fe640 1 -- 172.21.15.12:0/2722675085 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729856110 0 0) 0x7fd0880b47a0 con 0x7fd08815eae0 2024-04-15T15:22:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd087fff640 1 -- 172.21.15.12:0/2722675085 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0880b4d60 con 0x7fd0880d2360 2024-04-15T15:22:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.906+0000 7fd08ca24640 1 --2- 172.21.15.12:0/2722675085 >> v2:172.21.15.55:3300/0 conn(0x7fd08815e700 0x7fd0880b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.907+0000 7fd087fff640 1 --2- 172.21.15.12:0/2722675085 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880d2b60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0740097a0 tx=0x7fd07402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ec6bcda581830f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.907+0000 7fd0877fe640 1 -- 172.21.15.12:0/2722675085 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd074035020 con 0x7fd0880d2360 2024-04-15T15:22:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.907+0000 7fd0877fe640 1 -- 172.21.15.12:0/2722675085 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd074031830 con 0x7fd0880d2360 2024-04-15T15:22:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.907+0000 7fd0877fe640 1 -- 172.21.15.12:0/2722675085 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd074031c00 con 0x7fd0880d2360 2024-04-15T15:22:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.908+0000 7fd08ec88640 1 -- 172.21.15.12:0/2722675085 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 msgr2=0x7fd0880d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.908+0000 7fd08ec88640 1 --2- 172.21.15.12:0/2722675085 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880d2b60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0740097a0 tx=0x7fd07402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.908+0000 7fd08ec88640 1 -- 172.21.15.12:0/2722675085 shutdown_connections 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.908+0000 7fd08ec88640 1 --2- 172.21.15.12:0/2722675085 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880d2b60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.908+0000 7fd08ec88640 1 --2- 172.21.15.12:0/2722675085 >> v2:172.21.15.55:3300/0 conn(0x7fd08815e700 0x7fd0880b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.908+0000 7fd08ec88640 1 -- 172.21.15.12:0/2722675085 >> 172.21.15.12:0/2722675085 conn(0x7fd08808bf60 msgr2=0x7fd0880d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.908+0000 7fd08ec88640 1 -- 172.21.15.12:0/2722675085 shutdown_connections 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.908+0000 7fd08ec88640 1 -- 172.21.15.12:0/2722675085 wait complete. 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.909+0000 7fd08ec88640 1 Processor -- start 2024-04-15T15:22:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.909+0000 7fd08ec88640 1 -- start start 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08ec88640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08ec88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd08815e700 0x7fd088157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08ec88640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd08815eae0 0x7fd0881582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08ec88640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd0880b5d80 con 0x7fd08815eae0 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08ec88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd0880b4450 con 0x7fd08815e700 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08ec88640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd0880d1d40 con 0x7fd0880d2360 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08d225640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd08815eae0 0x7fd0881582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08ca24640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08d225640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd08815eae0 0x7fd0881582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49586/0 (socket says 172.21.15.12:49586) 2024-04-15T15:22:41.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd08d225640 1 -- 172.21.15.12:0/3365103681 learned_addr learned my addr 172.21.15.12:0/3365103681 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.910+0000 7fd087fff640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd08815e700 0x7fd088157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd08ca24640 1 -- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd08815e700 msgr2=0x7fd088157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd08ca24640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd08815e700 0x7fd088157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd08ca24640 1 -- 172.21.15.12:0/3365103681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd08815eae0 msgr2=0x7fd0881582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd08ca24640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd08815eae0 0x7fd0881582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd08ca24640 1 -- 172.21.15.12:0/3365103681 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd088160f80 con 0x7fd0880d2360 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd08ca24640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880ca890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00e900 tx=0x7fd07c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd0857fa640 1 -- 172.21.15.12:0/3365103681 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07c00ca00 con 0x7fd0880d2360 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd0857fa640 1 -- 172.21.15.12:0/3365103681 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd07c00cba0 con 0x7fd0880d2360 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd088161260 con 0x7fd0880d2360 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd0857fa640 1 -- 172.21.15.12:0/3365103681 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07c005990 con 0x7fd0880d2360 2024-04-15T15:22:41.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.911+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd088161820 con 0x7fd0880d2360 2024-04-15T15:22:41.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.912+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0880689e0 con 0x7fd0880d2360 2024-04-15T15:22:41.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.913+0000 7fd0857fa640 1 -- 172.21.15.12:0/3365103681 <== mon.2 v2:172.21.15.55:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd07c005b30 con 0x7fd0880d2360 2024-04-15T15:22:41.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.913+0000 7fd0857fa640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd06c06c8f0 0x7fd06c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:41.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.913+0000 7fd0857fa640 1 -- 172.21.15.12:0/3365103681 <== mon.2 v2:172.21.15.55:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fd07c08bff0 con 0x7fd0880d2360 2024-04-15T15:22:41.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.913+0000 7fd087fff640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd06c06c8f0 0x7fd06c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:41.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.914+0000 7fd087fff640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd06c06c8f0 0x7fd06c06edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd074036000 tx=0x7fd07403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:41.918+0000 7fd0857fa640 1 -- 172.21.15.12:0/3365103681 <== mon.2 v2:172.21.15.55:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd07c017020 con 0x7fd0880d2360 2024-04-15T15:22:42.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.100+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd0880ba500 con 0x7fd06c06c8f0 2024-04-15T15:22:42.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.107+0000 7fd0857fa640 1 -- 172.21.15.12:0/3365103681 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251959 (secure 0 0 0) 0x7fd074088850 con 0x7fd06c06c8f0 2024-04-15T15:22:42.112 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:42.113 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:22:42.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd06c06c8f0 msgr2=0x7fd06c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd06c06c8f0 0x7fd06c06edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd074036000 tx=0x7fd07403d040 comp rx=0 tx=0).stop 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 msgr2=0x7fd0880ca890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880ca890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00e900 tx=0x7fd07c00edc0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 shutdown_connections 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd08815eae0 0x7fd0881582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd06c06c8f0 0x7fd06c06edb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd08815e700 0x7fd088157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 --2- 172.21.15.12:0/3365103681 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd0880d2360 0x7fd0880ca890 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 >> 172.21.15.12:0/3365103681 conn(0x7fd08808bf60 msgr2=0x7fd0880d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 shutdown_connections 2024-04-15T15:22:42.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.110+0000 7fd08ec88640 1 -- 172.21.15.12:0/3365103681 wait complete. 2024-04-15T15:22:42.123 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-04-15T15:22:40.609506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237584,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737136,"statfs":{"total":773094113280,"available":772850827264,"internally_reserved":0,"allocated":4857856,"data_stored":2745804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002170"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295604+0000","last_change":"2024-04-15T15:22:33.295313+0000","last_active":"2024-04-15T15:22:34.295604+0000","last_peered":"2024-04-15T15:22:34.295604+0000","last_clean":"2024-04-15T15:22:34.295604+0000","last_became_active":"2024-04-15T15:22:33.295019+0000","last_became_peered":"2024-04-15T15:22:33.295019+0000","last_unstale":"2024-04-15T15:22:34.295604+0000","last_undegraded":"2024-04-15T15:22:34.295604+0000","last_fullsized":"2024-04-15T15:22:34.295604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:46:17.698031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295342+0000","last_change":"2024-04-15T15:22:33.294677+0000","last_active":"2024-04-15T15:22:34.295342+0000","last_peered":"2024-04-15T15:22:34.295342+0000","last_clean":"2024-04-15T15:22:34.295342+0000","last_became_active":"2024-04-15T15:22:33.294383+0000","last_became_peered":"2024-04-15T15:22:33.294383+0000","last_unstale":"2024-04-15T15:22:34.295342+0000","last_undegraded":"2024-04-15T15:22:34.295342+0000","last_fullsized":"2024-04-15T15:22:34.295342+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:47.415134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565304+0000","last_change":"2024-04-15T15:22:33.308305+0000","last_active":"2024-04-15T15:22:34.565304+0000","last_peered":"2024-04-15T15:22:34.565304+0000","last_clean":"2024-04-15T15:22:34.565304+0000","last_became_active":"2024-04-15T15:22:33.308000+0000","last_became_peered":"2024-04-15T15:22:33.308000+0000","last_unstale":"2024-04-15T15:22:34.565304+0000","last_undegraded":"2024-04-15T15:22:34.565304+0000","last_fullsized":"2024-04-15T15:22:34.565304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:30:58.162485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039672+0000","last_change":"2024-04-15T15:22:33.305255+0000","last_active":"2024-04-15T15:22:35.039672+0000","last_peered":"2024-04-15T15:22:35.039672+0000","last_clean":"2024-04-15T15:22:35.039672+0000","last_became_active":"2024-04-15T15:22:33.304994+0000","last_became_peered":"2024-04-15T15:22:33.304994+0000","last_unstale":"2024-04-15T15:22:35.039672+0000","last_undegraded":"2024-04-15T15:22:35.039672+0000","last_fullsized":"2024-04-15T15:22:35.039672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:28.120264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039279+0000","last_change":"2024-04-15T15:22:33.304553+0000","last_active":"2024-04-15T15:22:35.039279+0000","last_peered":"2024-04-15T15:22:35.039279+0000","last_clean":"2024-04-15T15:22:35.039279+0000","last_became_active":"2024-04-15T15:22:33.304200+0000","last_became_peered":"2024-04-15T15:22:33.304200+0000","last_unstale":"2024-04-15T15:22:35.039279+0000","last_undegraded":"2024-04-15T15:22:35.039279+0000","last_fullsized":"2024-04-15T15:22:35.039279+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:48.069224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039099+0000","last_change":"2024-04-15T15:22:33.296731+0000","last_active":"2024-04-15T15:22:35.039099+0000","last_peered":"2024-04-15T15:22:35.039099+0000","last_clean":"2024-04-15T15:22:35.039099+0000","last_became_active":"2024-04-15T15:22:33.295992+0000","last_became_peered":"2024-04-15T15:22:33.295992+0000","last_unstale":"2024-04-15T15:22:35.039099+0000","last_undegraded":"2024-04-15T15:22:35.039099+0000","last_fullsized":"2024-04-15T15:22:35.039099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:25:09.987700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305511+0000","last_change":"2024-04-15T15:22:33.298931+0000","last_active":"2024-04-15T15:22:34.305511+0000","last_peered":"2024-04-15T15:22:34.305511+0000","last_clean":"2024-04-15T15:22:34.305511+0000","last_became_active":"2024-04-15T15:22:33.298626+0000","last_became_peered":"2024-04-15T15:22:33.298626+0000","last_unstale":"2024-04-15T15:22:34.305511+0000","last_undegraded":"2024-04-15T15:22:34.305511+0000","last_fullsized":"2024-04-15T15:22:34.305511+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:33.588528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295124+0000","last_change":"2024-04-15T15:22:33.307911+0000","last_active":"2024-04-15T15:22:34.295124+0000","last_peered":"2024-04-15T15:22:34.295124+0000","last_clean":"2024-04-15T15:22:34.295124+0000","last_became_active":"2024-04-15T15:22:33.307643+0000","last_became_peered":"2024-04-15T15:22:33.307643+0000","last_unstale":"2024-04-15T15:22:34.295124+0000","last_undegraded":"2024-04-15T15:22:34.295124+0000","last_fullsized":"2024-04-15T15:22:34.295124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:11:33.617468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565545+0000","last_change":"2024-04-15T15:22:33.313825+0000","last_active":"2024-04-15T15:22:34.565545+0000","last_peered":"2024-04-15T15:22:34.565545+0000","last_clean":"2024-04-15T15:22:34.565545+0000","last_became_active":"2024-04-15T15:22:33.313619+0000","last_became_peered":"2024-04-15T15:22:33.313619+0000","last_unstale":"2024-04-15T15:22:34.565545+0000","last_undegraded":"2024-04-15T15:22:34.565545+0000","last_fullsized":"2024-04-15T15:22:34.565545+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:53:19.613211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305991+0000","last_change":"2024-04-15T15:22:33.314412+0000","last_active":"2024-04-15T15:22:34.305991+0000","last_peered":"2024-04-15T15:22:34.305991+0000","last_clean":"2024-04-15T15:22:34.305991+0000","last_became_active":"2024-04-15T15:22:33.313558+0000","last_became_peered":"2024-04-15T15:22:33.313558+0000","last_unstale":"2024-04-15T15:22:34.305991+0000","last_undegraded":"2024-04-15T15:22:34.305991+0000","last_fullsized":"2024-04-15T15:22:34.305991+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:26:40.045991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306181+0000","last_change":"2024-04-15T15:22:33.306879+0000","last_active":"2024-04-15T15:22:34.306181+0000","last_peered":"2024-04-15T15:22:34.306181+0000","last_clean":"2024-04-15T15:22:34.306181+0000","last_became_active":"2024-04-15T15:22:33.306416+0000","last_became_peered":"2024-04-15T15:22:33.306416+0000","last_unstale":"2024-04-15T15:22:34.306181+0000","last_undegraded":"2024-04-15T15:22:34.306181+0000","last_fullsized":"2024-04-15T15:22:34.306181+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:52.844126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565399+0000","last_change":"2024-04-15T15:22:33.298571+0000","last_active":"2024-04-15T15:22:34.565399+0000","last_peered":"2024-04-15T15:22:34.565399+0000","last_clean":"2024-04-15T15:22:34.565399+0000","last_became_active":"2024-04-15T15:22:33.298254+0000","last_became_peered":"2024-04-15T15:22:33.298254+0000","last_unstale":"2024-04-15T15:22:34.565399+0000","last_undegraded":"2024-04-15T15:22:34.565399+0000","last_fullsized":"2024-04-15T15:22:34.565399+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:47:45.911804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305719+0000","last_change":"2024-04-15T15:22:33.313702+0000","last_active":"2024-04-15T15:22:34.305719+0000","last_peered":"2024-04-15T15:22:34.305719+0000","last_clean":"2024-04-15T15:22:34.305719+0000","last_became_active":"2024-04-15T15:22:33.313391+0000","last_became_peered":"2024-04-15T15:22:33.313391+0000","last_unstale":"2024-04-15T15:22:34.305719+0000","last_undegraded":"2024-04-15T15:22:34.305719+0000","last_fullsized":"2024-04-15T15:22:34.305719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:47.415134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538077+0000","last_change":"2024-04-15T15:22:33.301615+0000","last_active":"2024-04-15T15:22:34.538077+0000","last_peered":"2024-04-15T15:22:34.538077+0000","last_clean":"2024-04-15T15:22:34.538077+0000","last_became_active":"2024-04-15T15:22:33.301298+0000","last_became_peered":"2024-04-15T15:22:33.301298+0000","last_unstale":"2024-04-15T15:22:34.538077+0000","last_undegraded":"2024-04-15T15:22:34.538077+0000","last_fullsized":"2024-04-15T15:22:34.538077+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:53:39.518417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305528+0000","last_change":"2024-04-15T15:22:33.300507+0000","last_active":"2024-04-15T15:22:34.305528+0000","last_peered":"2024-04-15T15:22:34.305528+0000","last_clean":"2024-04-15T15:22:34.305528+0000","last_became_active":"2024-04-15T15:22:33.300105+0000","last_became_peered":"2024-04-15T15:22:33.300105+0000","last_unstale":"2024-04-15T15:22:34.305528+0000","last_undegraded":"2024-04-15T15:22:34.305528+0000","last_fullsized":"2024-04-15T15:22:34.305528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:26.586364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705033+0000","last_change":"2024-04-15T15:22:33.314263+0000","last_active":"2024-04-15T15:22:34.705033+0000","last_peered":"2024-04-15T15:22:34.705033+0000","last_clean":"2024-04-15T15:22:34.705033+0000","last_became_active":"2024-04-15T15:22:33.313829+0000","last_became_peered":"2024-04-15T15:22:33.313829+0000","last_unstale":"2024-04-15T15:22:34.705033+0000","last_undegraded":"2024-04-15T15:22:34.705033+0000","last_fullsized":"2024-04-15T15:22:34.705033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:51:51.191240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705498+0000","last_change":"2024-04-15T15:22:33.314419+0000","last_active":"2024-04-15T15:22:34.705498+0000","last_peered":"2024-04-15T15:22:34.705498+0000","last_clean":"2024-04-15T15:22:34.705498+0000","last_became_active":"2024-04-15T15:22:33.313923+0000","last_became_peered":"2024-04-15T15:22:33.313923+0000","last_unstale":"2024-04-15T15:22:34.705498+0000","last_undegraded":"2024-04-15T15:22:34.705498+0000","last_fullsized":"2024-04-15T15:22:34.705498+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:47.415134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038948+0000","last_change":"2024-04-15T15:22:33.305542+0000","last_active":"2024-04-15T15:22:35.038948+0000","last_peered":"2024-04-15T15:22:35.038948+0000","last_clean":"2024-04-15T15:22:35.038948+0000","last_became_active":"2024-04-15T15:22:33.305231+0000","last_became_peered":"2024-04-15T15:22:33.305231+0000","last_unstale":"2024-04-15T15:22:35.038948+0000","last_undegraded":"2024-04-15T15:22:35.038948+0000","last_fullsized":"2024-04-15T15:22:35.038948+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:18:35.877536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039548+0000","last_change":"2024-04-15T15:22:33.302274+0000","last_active":"2024-04-15T15:22:35.039548+0000","last_peered":"2024-04-15T15:22:35.039548+0000","last_clean":"2024-04-15T15:22:35.039548+0000","last_became_active":"2024-04-15T15:22:33.301981+0000","last_became_peered":"2024-04-15T15:22:33.301981+0000","last_unstale":"2024-04-15T15:22:35.039548+0000","last_undegraded":"2024-04-15T15:22:35.039548+0000","last_fullsized":"2024-04-15T15:22:35.039548+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:21:14.004136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038970+0000","last_change":"2024-04-15T15:22:33.305686+0000","last_active":"2024-04-15T15:22:35.038970+0000","last_peered":"2024-04-15T15:22:35.038970+0000","last_clean":"2024-04-15T15:22:35.038970+0000","last_became_active":"2024-04-15T15:22:33.305390+0000","last_became_peered":"2024-04-15T15:22:33.305390+0000","last_unstale":"2024-04-15T15:22:35.038970+0000","last_undegraded":"2024-04-15T15:22:35.038970+0000","last_fullsized":"2024-04-15T15:22:35.038970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:53:39.518417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305803+0000","last_change":"2024-04-15T15:22:33.301798+0000","last_active":"2024-04-15T15:22:34.305803+0000","last_peered":"2024-04-15T15:22:34.305803+0000","last_clean":"2024-04-15T15:22:34.305803+0000","last_became_active":"2024-04-15T15:22:33.301513+0000","last_became_peered":"2024-04-15T15:22:33.301513+0000","last_unstale":"2024-04-15T15:22:34.305803+0000","last_undegraded":"2024-04-15T15:22:34.305803+0000","last_fullsized":"2024-04-15T15:22:34.305803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:51:51.191240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538355+0000","last_change":"2024-04-15T15:22:33.302535+0000","last_active":"2024-04-15T15:22:34.538355+0000","last_peered":"2024-04-15T15:22:34.538355+0000","last_clean":"2024-04-15T15:22:34.538355+0000","last_became_active":"2024-04-15T15:22:33.301667+0000","last_became_peered":"2024-04-15T15:22:33.301667+0000","last_unstale":"2024-04-15T15:22:34.538355+0000","last_undegraded":"2024-04-15T15:22:34.538355+0000","last_fullsized":"2024-04-15T15:22:34.538355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:18:22.291418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705072+0000","last_change":"2024-04-15T15:22:33.308787+0000","last_active":"2024-04-15T15:22:34.705072+0000","last_peered":"2024-04-15T15:22:34.705072+0000","last_clean":"2024-04-15T15:22:34.705072+0000","last_became_active":"2024-04-15T15:22:33.308238+0000","last_became_peered":"2024-04-15T15:22:33.308238+0000","last_unstale":"2024-04-15T15:22:34.705072+0000","last_undegraded":"2024-04-15T15:22:34.705072+0000","last_fullsized":"2024-04-15T15:22:34.705072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:22:11.084708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039850+0000","last_change":"2024-04-15T15:22:33.304264+0000","last_active":"2024-04-15T15:22:35.039850+0000","last_peered":"2024-04-15T15:22:35.039850+0000","last_clean":"2024-04-15T15:22:35.039850+0000","last_became_active":"2024-04-15T15:22:33.303941+0000","last_became_peered":"2024-04-15T15:22:33.303941+0000","last_unstale":"2024-04-15T15:22:35.039850+0000","last_undegraded":"2024-04-15T15:22:35.039850+0000","last_fullsized":"2024-04-15T15:22:35.039850+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:02:15.691472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038908+0000","last_change":"2024-04-15T15:22:33.297937+0000","last_active":"2024-04-15T15:22:35.038908+0000","last_peered":"2024-04-15T15:22:35.038908+0000","last_clean":"2024-04-15T15:22:35.038908+0000","last_became_active":"2024-04-15T15:22:33.297425+0000","last_became_peered":"2024-04-15T15:22:33.297425+0000","last_unstale":"2024-04-15T15:22:35.038908+0000","last_undegraded":"2024-04-15T15:22:35.038908+0000","last_fullsized":"2024-04-15T15:22:35.038908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:17:29.189541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039595+0000","last_change":"2024-04-15T15:22:33.301393+0000","last_active":"2024-04-15T15:22:35.039595+0000","last_peered":"2024-04-15T15:22:35.039595+0000","last_clean":"2024-04-15T15:22:35.039595+0000","last_became_active":"2024-04-15T15:22:33.299906+0000","last_became_peered":"2024-04-15T15:22:33.299906+0000","last_unstale":"2024-04-15T15:22:35.039595+0000","last_undegraded":"2024-04-15T15:22:35.039595+0000","last_fullsized":"2024-04-15T15:22:35.039595+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:30:58.162485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038906+0000","last_change":"2024-04-15T15:22:33.298932+0000","last_active":"2024-04-15T15:22:35.038906+0000","last_peered":"2024-04-15T15:22:35.038906+0000","last_clean":"2024-04-15T15:22:35.038906+0000","last_became_active":"2024-04-15T15:22:33.298633+0000","last_became_peered":"2024-04-15T15:22:33.298633+0000","last_unstale":"2024-04-15T15:22:35.038906+0000","last_undegraded":"2024-04-15T15:22:35.038906+0000","last_fullsized":"2024-04-15T15:22:35.038906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:02:15.691472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305804+0000","last_change":"2024-04-15T15:22:33.302161+0000","last_active":"2024-04-15T15:22:34.305804+0000","last_peered":"2024-04-15T15:22:34.305804+0000","last_clean":"2024-04-15T15:22:34.305804+0000","last_became_active":"2024-04-15T15:22:33.299226+0000","last_became_peered":"2024-04-15T15:22:33.299226+0000","last_unstale":"2024-04-15T15:22:34.305804+0000","last_undegraded":"2024-04-15T15:22:34.305804+0000","last_fullsized":"2024-04-15T15:22:34.305804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:18.339496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038719+0000","last_change":"2024-04-15T15:22:33.300377+0000","last_active":"2024-04-15T15:22:35.038719+0000","last_peered":"2024-04-15T15:22:35.038719+0000","last_clean":"2024-04-15T15:22:35.038719+0000","last_became_active":"2024-04-15T15:22:33.300088+0000","last_became_peered":"2024-04-15T15:22:33.300088+0000","last_unstale":"2024-04-15T15:22:35.038719+0000","last_undegraded":"2024-04-15T15:22:35.038719+0000","last_fullsized":"2024-04-15T15:22:35.038719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:46.712200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038706+0000","last_change":"2024-04-15T15:22:33.298247+0000","last_active":"2024-04-15T15:22:35.038706+0000","last_peered":"2024-04-15T15:22:35.038706+0000","last_clean":"2024-04-15T15:22:35.038706+0000","last_became_active":"2024-04-15T15:22:33.297608+0000","last_became_peered":"2024-04-15T15:22:33.297608+0000","last_unstale":"2024-04-15T15:22:35.038706+0000","last_undegraded":"2024-04-15T15:22:35.038706+0000","last_fullsized":"2024-04-15T15:22:35.038706+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:48.069224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705123+0000","last_change":"2024-04-15T15:22:33.308795+0000","last_active":"2024-04-15T15:22:34.705123+0000","last_peered":"2024-04-15T15:22:34.705123+0000","last_clean":"2024-04-15T15:22:34.705123+0000","last_became_active":"2024-04-15T15:22:33.308527+0000","last_became_peered":"2024-04-15T15:22:33.308527+0000","last_unstale":"2024-04-15T15:22:34.705123+0000","last_undegraded":"2024-04-15T15:22:34.705123+0000","last_fullsized":"2024-04-15T15:22:34.705123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:53:19.613211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038996+0000","last_change":"2024-04-15T15:22:33.301064+0000","last_active":"2024-04-15T15:22:35.038996+0000","last_peered":"2024-04-15T15:22:35.038996+0000","last_clean":"2024-04-15T15:22:35.038996+0000","last_became_active":"2024-04-15T15:22:33.299748+0000","last_became_peered":"2024-04-15T15:22:33.299748+0000","last_unstale":"2024-04-15T15:22:35.038996+0000","last_undegraded":"2024-04-15T15:22:35.038996+0000","last_fullsized":"2024-04-15T15:22:35.038996+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:14:54.794343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705249+0000","last_change":"2024-04-15T15:22:33.314561+0000","last_active":"2024-04-15T15:22:34.705249+0000","last_peered":"2024-04-15T15:22:34.705249+0000","last_clean":"2024-04-15T15:22:34.705249+0000","last_became_active":"2024-04-15T15:22:33.314040+0000","last_became_peered":"2024-04-15T15:22:33.314040+0000","last_unstale":"2024-04-15T15:22:34.705249+0000","last_undegraded":"2024-04-15T15:22:34.705249+0000","last_fullsized":"2024-04-15T15:22:34.705249+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:26.586364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038894+0000","last_change":"2024-04-15T15:22:32.295369+0000","last_active":"2024-04-15T15:22:35.038894+0000","last_peered":"2024-04-15T15:22:35.038894+0000","last_clean":"2024-04-15T15:22:35.038894+0000","last_became_active":"2024-04-15T15:22:32.295054+0000","last_became_peered":"2024-04-15T15:22:32.295054+0000","last_unstale":"2024-04-15T15:22:35.038894+0000","last_undegraded":"2024-04-15T15:22:35.038894+0000","last_fullsized":"2024-04-15T15:22:35.038894+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039729+0000","last_change":"2024-04-15T15:22:33.311968+0000","last_active":"2024-04-15T15:22:35.039729+0000","last_peered":"2024-04-15T15:22:35.039729+0000","last_clean":"2024-04-15T15:22:35.039729+0000","last_became_active":"2024-04-15T15:22:33.311690+0000","last_became_peered":"2024-04-15T15:22:33.311690+0000","last_unstale":"2024-04-15T15:22:35.039729+0000","last_undegraded":"2024-04-15T15:22:35.039729+0000","last_fullsized":"2024-04-15T15:22:35.039729+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:49:13.345861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565472+0000","last_change":"2024-04-15T15:22:32.298663+0000","last_active":"2024-04-15T15:22:34.565472+0000","last_peered":"2024-04-15T15:22:34.565472+0000","last_clean":"2024-04-15T15:22:34.565472+0000","last_became_active":"2024-04-15T15:22:32.298296+0000","last_became_peered":"2024-04-15T15:22:32.298296+0000","last_unstale":"2024-04-15T15:22:34.565472+0000","last_undegraded":"2024-04-15T15:22:34.565472+0000","last_fullsized":"2024-04-15T15:22:34.565472+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538568+0000","last_change":"2024-04-15T15:22:33.296800+0000","last_active":"2024-04-15T15:22:34.538568+0000","last_peered":"2024-04-15T15:22:34.538568+0000","last_clean":"2024-04-15T15:22:34.538568+0000","last_became_active":"2024-04-15T15:22:33.296491+0000","last_became_peered":"2024-04-15T15:22:33.296491+0000","last_unstale":"2024-04-15T15:22:34.538568+0000","last_undegraded":"2024-04-15T15:22:34.538568+0000","last_fullsized":"2024-04-15T15:22:34.538568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:30:58.162485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039847+0000","last_change":"2024-04-15T15:22:32.301621+0000","last_active":"2024-04-15T15:22:35.039847+0000","last_peered":"2024-04-15T15:22:35.039847+0000","last_clean":"2024-04-15T15:22:35.039847+0000","last_became_active":"2024-04-15T15:22:32.301253+0000","last_became_peered":"2024-04-15T15:22:32.301253+0000","last_unstale":"2024-04-15T15:22:35.039847+0000","last_undegraded":"2024-04-15T15:22:35.039847+0000","last_fullsized":"2024-04-15T15:22:35.039847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295542+0000","last_change":"2024-04-15T15:22:33.296469+0000","last_active":"2024-04-15T15:22:34.295542+0000","last_peered":"2024-04-15T15:22:34.295542+0000","last_clean":"2024-04-15T15:22:34.295542+0000","last_became_active":"2024-04-15T15:22:33.296158+0000","last_became_peered":"2024-04-15T15:22:33.296158+0000","last_unstale":"2024-04-15T15:22:34.295542+0000","last_undegraded":"2024-04-15T15:22:34.295542+0000","last_fullsized":"2024-04-15T15:22:34.295542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:51:51.191240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538325+0000","last_change":"2024-04-15T15:22:32.297657+0000","last_active":"2024-04-15T15:22:34.538325+0000","last_peered":"2024-04-15T15:22:34.538325+0000","last_clean":"2024-04-15T15:22:34.538325+0000","last_became_active":"2024-04-15T15:22:32.297419+0000","last_became_peered":"2024-04-15T15:22:32.297419+0000","last_unstale":"2024-04-15T15:22:34.538325+0000","last_undegraded":"2024-04-15T15:22:34.538325+0000","last_fullsized":"2024-04-15T15:22:34.538325+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:54:31.073202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565200+0000","last_change":"2024-04-15T15:22:33.300618+0000","last_active":"2024-04-15T15:22:34.565200+0000","last_peered":"2024-04-15T15:22:34.565200+0000","last_clean":"2024-04-15T15:22:34.565200+0000","last_became_active":"2024-04-15T15:22:33.300356+0000","last_became_peered":"2024-04-15T15:22:33.300356+0000","last_unstale":"2024-04-15T15:22:34.565200+0000","last_undegraded":"2024-04-15T15:22:34.565200+0000","last_fullsized":"2024-04-15T15:22:34.565200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:25:09.987700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305834+0000","last_change":"2024-04-15T15:22:32.293808+0000","last_active":"2024-04-15T15:22:34.305834+0000","last_peered":"2024-04-15T15:22:34.305834+0000","last_clean":"2024-04-15T15:22:34.305834+0000","last_became_active":"2024-04-15T15:22:32.293474+0000","last_became_peered":"2024-04-15T15:22:32.293474+0000","last_unstale":"2024-04-15T15:22:34.305834+0000","last_undegraded":"2024-04-15T15:22:34.305834+0000","last_fullsized":"2024-04-15T15:22:34.305834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:02:14.685440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305640+0000","last_change":"2024-04-15T15:22:33.305485+0000","last_active":"2024-04-15T15:22:34.305640+0000","last_peered":"2024-04-15T15:22:34.305640+0000","last_clean":"2024-04-15T15:22:34.305640+0000","last_became_active":"2024-04-15T15:22:33.305123+0000","last_became_peered":"2024-04-15T15:22:33.305123+0000","last_unstale":"2024-04-15T15:22:34.305640+0000","last_undegraded":"2024-04-15T15:22:34.305640+0000","last_fullsized":"2024-04-15T15:22:34.305640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:48.797544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705020+0000","last_change":"2024-04-15T15:22:32.299916+0000","last_active":"2024-04-15T15:22:34.705020+0000","last_peered":"2024-04-15T15:22:34.705020+0000","last_clean":"2024-04-15T15:22:34.705020+0000","last_became_active":"2024-04-15T15:22:32.299576+0000","last_became_peered":"2024-04-15T15:22:32.299576+0000","last_unstale":"2024-04-15T15:22:34.705020+0000","last_undegraded":"2024-04-15T15:22:34.705020+0000","last_fullsized":"2024-04-15T15:22:34.705020+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538525+0000","last_change":"2024-04-15T15:22:33.296037+0000","last_active":"2024-04-15T15:22:34.538525+0000","last_peered":"2024-04-15T15:22:34.538525+0000","last_clean":"2024-04-15T15:22:34.538525+0000","last_became_active":"2024-04-15T15:22:33.295570+0000","last_became_peered":"2024-04-15T15:22:33.295570+0000","last_unstale":"2024-04-15T15:22:34.538525+0000","last_undegraded":"2024-04-15T15:22:34.538525+0000","last_fullsized":"2024-04-15T15:22:34.538525+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:42:06.582334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039045+0000","last_change":"2024-04-15T15:22:32.301232+0000","last_active":"2024-04-15T15:22:35.039045+0000","last_peered":"2024-04-15T15:22:35.039045+0000","last_clean":"2024-04-15T15:22:35.039045+0000","last_became_active":"2024-04-15T15:22:32.300885+0000","last_became_peered":"2024-04-15T15:22:32.300885+0000","last_unstale":"2024-04-15T15:22:35.039045+0000","last_undegraded":"2024-04-15T15:22:35.039045+0000","last_fullsized":"2024-04-15T15:22:35.039045+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295257+0000","last_change":"2024-04-15T15:22:33.305865+0000","last_active":"2024-04-15T15:22:34.295257+0000","last_peered":"2024-04-15T15:22:34.295257+0000","last_clean":"2024-04-15T15:22:34.295257+0000","last_became_active":"2024-04-15T15:22:33.305605+0000","last_became_peered":"2024-04-15T15:22:33.305605+0000","last_unstale":"2024-04-15T15:22:34.295257+0000","last_undegraded":"2024-04-15T15:22:34.295257+0000","last_fullsized":"2024-04-15T15:22:34.295257+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:17:04.067763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.537966+0000","last_change":"2024-04-15T15:22:32.288888+0000","last_active":"2024-04-15T15:22:34.537966+0000","last_peered":"2024-04-15T15:22:34.537966+0000","last_clean":"2024-04-15T15:22:34.537966+0000","last_became_active":"2024-04-15T15:22:32.288559+0000","last_became_peered":"2024-04-15T15:22:32.288559+0000","last_unstale":"2024-04-15T15:22:34.537966+0000","last_undegraded":"2024-04-15T15:22:34.537966+0000","last_fullsized":"2024-04-15T15:22:34.537966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:06.406224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039716+0000","last_change":"2024-04-15T15:22:33.296928+0000","last_active":"2024-04-15T15:22:35.039716+0000","last_peered":"2024-04-15T15:22:35.039716+0000","last_clean":"2024-04-15T15:22:35.039716+0000","last_became_active":"2024-04-15T15:22:33.296618+0000","last_became_peered":"2024-04-15T15:22:33.296618+0000","last_unstale":"2024-04-15T15:22:35.039716+0000","last_undegraded":"2024-04-15T15:22:35.039716+0000","last_fullsized":"2024-04-15T15:22:35.039716+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:46.712200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295636+0000","last_change":"2024-04-15T15:22:32.292353+0000","last_active":"2024-04-15T15:22:34.295636+0000","last_peered":"2024-04-15T15:22:34.295636+0000","last_clean":"2024-04-15T15:22:34.295636+0000","last_became_active":"2024-04-15T15:22:32.292043+0000","last_became_peered":"2024-04-15T15:22:32.292043+0000","last_unstale":"2024-04-15T15:22:34.295636+0000","last_undegraded":"2024-04-15T15:22:34.295636+0000","last_fullsized":"2024-04-15T15:22:34.295636+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:18:34.871504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705618+0000","last_change":"2024-04-15T15:22:33.309220+0000","last_active":"2024-04-15T15:22:34.705618+0000","last_peered":"2024-04-15T15:22:34.705618+0000","last_clean":"2024-04-15T15:22:34.705618+0000","last_became_active":"2024-04-15T15:22:33.308201+0000","last_became_peered":"2024-04-15T15:22:33.308201+0000","last_unstale":"2024-04-15T15:22:34.705618+0000","last_undegraded":"2024-04-15T15:22:34.705618+0000","last_fullsized":"2024-04-15T15:22:34.705618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:18.339496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306092+0000","last_change":"2024-04-15T15:22:32.294706+0000","last_active":"2024-04-15T15:22:34.306092+0000","last_peered":"2024-04-15T15:22:34.306092+0000","last_clean":"2024-04-15T15:22:34.306092+0000","last_became_active":"2024-04-15T15:22:32.294353+0000","last_became_peered":"2024-04-15T15:22:32.294353+0000","last_unstale":"2024-04-15T15:22:34.306092+0000","last_undegraded":"2024-04-15T15:22:34.306092+0000","last_fullsized":"2024-04-15T15:22:34.306092+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:53:38.512384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565590+0000","last_change":"2024-04-15T15:22:33.309489+0000","last_active":"2024-04-15T15:22:34.565590+0000","last_peered":"2024-04-15T15:22:34.565590+0000","last_clean":"2024-04-15T15:22:34.565590+0000","last_became_active":"2024-04-15T15:22:33.309241+0000","last_became_peered":"2024-04-15T15:22:33.309241+0000","last_unstale":"2024-04-15T15:22:34.565590+0000","last_undegraded":"2024-04-15T15:22:34.565590+0000","last_fullsized":"2024-04-15T15:22:34.565590+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:11:33.617468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306285+0000","last_change":"2024-04-15T15:22:32.294321+0000","last_active":"2024-04-15T15:22:34.306285+0000","last_peered":"2024-04-15T15:22:34.306285+0000","last_clean":"2024-04-15T15:22:34.306285+0000","last_became_active":"2024-04-15T15:22:32.294043+0000","last_became_peered":"2024-04-15T15:22:32.294043+0000","last_unstale":"2024-04-15T15:22:34.306285+0000","last_undegraded":"2024-04-15T15:22:34.306285+0000","last_fullsized":"2024-04-15T15:22:34.306285+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:47.063192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538601+0000","last_change":"2024-04-15T15:22:33.302287+0000","last_active":"2024-04-15T15:22:34.538601+0000","last_peered":"2024-04-15T15:22:34.538601+0000","last_clean":"2024-04-15T15:22:34.538601+0000","last_became_active":"2024-04-15T15:22:33.302020+0000","last_became_peered":"2024-04-15T15:22:33.302020+0000","last_unstale":"2024-04-15T15:22:34.538601+0000","last_undegraded":"2024-04-15T15:22:34.538601+0000","last_fullsized":"2024-04-15T15:22:34.538601+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:49:13.345861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295609+0000","last_change":"2024-04-15T15:22:32.298894+0000","last_active":"2024-04-15T15:22:34.295609+0000","last_peered":"2024-04-15T15:22:34.295609+0000","last_clean":"2024-04-15T15:22:34.295609+0000","last_became_active":"2024-04-15T15:22:32.298534+0000","last_became_peered":"2024-04-15T15:22:32.298534+0000","last_unstale":"2024-04-15T15:22:34.295609+0000","last_undegraded":"2024-04-15T15:22:34.295609+0000","last_fullsized":"2024-04-15T15:22:34.295609+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:17:28.183508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565266+0000","last_change":"2024-04-15T15:22:33.308028+0000","last_active":"2024-04-15T15:22:34.565266+0000","last_peered":"2024-04-15T15:22:34.565266+0000","last_clean":"2024-04-15T15:22:34.565266+0000","last_became_active":"2024-04-15T15:22:33.307745+0000","last_became_peered":"2024-04-15T15:22:33.307745+0000","last_unstale":"2024-04-15T15:22:34.565266+0000","last_undegraded":"2024-04-15T15:22:34.565266+0000","last_fullsized":"2024-04-15T15:22:34.565266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:42:06.582334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295608+0000","last_change":"2024-04-15T15:22:32.287177+0000","last_active":"2024-04-15T15:22:34.295608+0000","last_peered":"2024-04-15T15:22:34.295608+0000","last_clean":"2024-04-15T15:22:34.295608+0000","last_became_active":"2024-04-15T15:22:32.286887+0000","last_became_peered":"2024-04-15T15:22:32.286887+0000","last_unstale":"2024-04-15T15:22:34.295608+0000","last_undegraded":"2024-04-15T15:22:34.295608+0000","last_fullsized":"2024-04-15T15:22:34.295608+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:02:14.685440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295451+0000","last_change":"2024-04-15T15:22:33.294006+0000","last_active":"2024-04-15T15:22:34.295451+0000","last_peered":"2024-04-15T15:22:34.295451+0000","last_clean":"2024-04-15T15:22:34.295451+0000","last_became_active":"2024-04-15T15:22:33.293049+0000","last_became_peered":"2024-04-15T15:22:33.293049+0000","last_unstale":"2024-04-15T15:22:34.295451+0000","last_undegraded":"2024-04-15T15:22:34.295451+0000","last_fullsized":"2024-04-15T15:22:34.295451+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:48.797544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039203+0000","last_change":"2024-04-15T15:22:32.301053+0000","last_active":"2024-04-15T15:22:35.039203+0000","last_peered":"2024-04-15T15:22:35.039203+0000","last_clean":"2024-04-15T15:22:35.039203+0000","last_became_active":"2024-04-15T15:22:32.300755+0000","last_became_peered":"2024-04-15T15:22:32.300755+0000","last_unstale":"2024-04-15T15:22:35.039203+0000","last_undegraded":"2024-04-15T15:22:35.039203+0000","last_fullsized":"2024-04-15T15:22:35.039203+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705129+0000","last_change":"2024-04-15T15:22:33.307952+0000","last_active":"2024-04-15T15:22:34.705129+0000","last_peered":"2024-04-15T15:22:34.705129+0000","last_clean":"2024-04-15T15:22:34.705129+0000","last_became_active":"2024-04-15T15:22:33.307634+0000","last_became_peered":"2024-04-15T15:22:33.307634+0000","last_unstale":"2024-04-15T15:22:34.705129+0000","last_undegraded":"2024-04-15T15:22:34.705129+0000","last_fullsized":"2024-04-15T15:22:34.705129+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:11:33.617468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038515+0000","last_change":"2024-04-15T15:22:32.293921+0000","last_active":"2024-04-15T15:22:35.038515+0000","last_peered":"2024-04-15T15:22:35.038515+0000","last_clean":"2024-04-15T15:22:35.038515+0000","last_became_active":"2024-04-15T15:22:32.293641+0000","last_became_peered":"2024-04-15T15:22:32.293641+0000","last_unstale":"2024-04-15T15:22:35.038515+0000","last_undegraded":"2024-04-15T15:22:35.038515+0000","last_fullsized":"2024-04-15T15:22:35.038515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:09:33.653846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295341+0000","last_change":"2024-04-15T15:22:33.292750+0000","last_active":"2024-04-15T15:22:34.295341+0000","last_peered":"2024-04-15T15:22:34.295341+0000","last_clean":"2024-04-15T15:22:34.295341+0000","last_became_active":"2024-04-15T15:22:33.292441+0000","last_became_peered":"2024-04-15T15:22:33.292441+0000","last_unstale":"2024-04-15T15:22:34.295341+0000","last_undegraded":"2024-04-15T15:22:34.295341+0000","last_fullsized":"2024-04-15T15:22:34.295341+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:26.586364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705012+0000","last_change":"2024-04-15T15:22:32.299339+0000","last_active":"2024-04-15T15:22:34.705012+0000","last_peered":"2024-04-15T15:22:34.705012+0000","last_clean":"2024-04-15T15:22:34.705012+0000","last_became_active":"2024-04-15T15:22:32.298921+0000","last_became_peered":"2024-04-15T15:22:32.298921+0000","last_unstale":"2024-04-15T15:22:34.705012+0000","last_undegraded":"2024-04-15T15:22:34.705012+0000","last_fullsized":"2024-04-15T15:22:34.705012+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:21:12.998104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295625+0000","last_change":"2024-04-15T15:22:33.296963+0000","last_active":"2024-04-15T15:22:34.295625+0000","last_peered":"2024-04-15T15:22:34.295625+0000","last_clean":"2024-04-15T15:22:34.295625+0000","last_became_active":"2024-04-15T15:22:33.296639+0000","last_became_peered":"2024-04-15T15:22:33.296639+0000","last_unstale":"2024-04-15T15:22:34.295625+0000","last_undegraded":"2024-04-15T15:22:34.295625+0000","last_fullsized":"2024-04-15T15:22:34.295625+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:26:40.045991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306207+0000","last_change":"2024-04-15T15:22:32.299157+0000","last_active":"2024-04-15T15:22:34.306207+0000","last_peered":"2024-04-15T15:22:34.306207+0000","last_clean":"2024-04-15T15:22:34.306207+0000","last_became_active":"2024-04-15T15:22:32.298836+0000","last_became_peered":"2024-04-15T15:22:32.298836+0000","last_unstale":"2024-04-15T15:22:34.306207+0000","last_undegraded":"2024-04-15T15:22:34.306207+0000","last_fullsized":"2024-04-15T15:22:34.306207+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:17:28.183508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306413+0000","last_change":"2024-04-15T15:22:33.305470+0000","last_active":"2024-04-15T15:22:34.306413+0000","last_peered":"2024-04-15T15:22:34.306413+0000","last_clean":"2024-04-15T15:22:34.306413+0000","last_became_active":"2024-04-15T15:22:33.305182+0000","last_became_peered":"2024-04-15T15:22:33.305182+0000","last_unstale":"2024-04-15T15:22:34.306413+0000","last_undegraded":"2024-04-15T15:22:34.306413+0000","last_fullsized":"2024-04-15T15:22:34.306413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:40:22.473471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538605+0000","last_change":"2024-04-15T15:22:32.301976+0000","last_active":"2024-04-15T15:22:34.538605+0000","last_peered":"2024-04-15T15:22:34.538605+0000","last_clean":"2024-04-15T15:22:34.538605+0000","last_became_active":"2024-04-15T15:22:32.301692+0000","last_became_peered":"2024-04-15T15:22:32.301692+0000","last_unstale":"2024-04-15T15:22:34.538605+0000","last_undegraded":"2024-04-15T15:22:34.538605+0000","last_fullsized":"2024-04-15T15:22:34.538605+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538486+0000","last_change":"2024-04-15T15:22:33.304545+0000","last_active":"2024-04-15T15:22:34.538486+0000","last_peered":"2024-04-15T15:22:34.538486+0000","last_clean":"2024-04-15T15:22:34.538486+0000","last_became_active":"2024-04-15T15:22:33.304196+0000","last_became_peered":"2024-04-15T15:22:33.304196+0000","last_unstale":"2024-04-15T15:22:34.538486+0000","last_undegraded":"2024-04-15T15:22:34.538486+0000","last_fullsized":"2024-04-15T15:22:34.538486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:25:09.987700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039681+0000","last_change":"2024-04-15T15:22:32.299415+0000","last_active":"2024-04-15T15:22:35.039681+0000","last_peered":"2024-04-15T15:22:35.039681+0000","last_clean":"2024-04-15T15:22:35.039681+0000","last_became_active":"2024-04-15T15:22:32.299078+0000","last_became_peered":"2024-04-15T15:22:32.299078+0000","last_unstale":"2024-04-15T15:22:35.039681+0000","last_undegraded":"2024-04-15T15:22:35.039681+0000","last_fullsized":"2024-04-15T15:22:35.039681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:05:16.011118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039501+0000","last_change":"2024-04-15T15:22:33.291429+0000","last_active":"2024-04-15T15:22:35.039501+0000","last_peered":"2024-04-15T15:22:35.039501+0000","last_clean":"2024-04-15T15:22:35.039501+0000","last_became_active":"2024-04-15T15:22:33.291151+0000","last_became_peered":"2024-04-15T15:22:33.291151+0000","last_unstale":"2024-04-15T15:22:35.039501+0000","last_undegraded":"2024-04-15T15:22:35.039501+0000","last_fullsized":"2024-04-15T15:22:35.039501+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:47.554681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039454+0000","last_change":"2024-04-15T15:22:32.301462+0000","last_active":"2024-04-15T15:22:35.039454+0000","last_peered":"2024-04-15T15:22:35.039454+0000","last_clean":"2024-04-15T15:22:35.039454+0000","last_became_active":"2024-04-15T15:22:32.301176+0000","last_became_peered":"2024-04-15T15:22:32.301176+0000","last_unstale":"2024-04-15T15:22:35.039454+0000","last_undegraded":"2024-04-15T15:22:35.039454+0000","last_fullsized":"2024-04-15T15:22:35.039454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:49:46.548649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705360+0000","last_change":"2024-04-15T15:22:33.309028+0000","last_active":"2024-04-15T15:22:34.705360+0000","last_peered":"2024-04-15T15:22:34.705360+0000","last_clean":"2024-04-15T15:22:34.705360+0000","last_became_active":"2024-04-15T15:22:33.308424+0000","last_became_peered":"2024-04-15T15:22:33.308424+0000","last_unstale":"2024-04-15T15:22:34.705360+0000","last_undegraded":"2024-04-15T15:22:34.705360+0000","last_fullsized":"2024-04-15T15:22:34.705360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:33.588528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039242+0000","last_change":"2024-04-15T15:22:32.290430+0000","last_active":"2024-04-15T15:22:35.039242+0000","last_peered":"2024-04-15T15:22:35.039242+0000","last_clean":"2024-04-15T15:22:35.039242+0000","last_became_active":"2024-04-15T15:22:32.290116+0000","last_became_peered":"2024-04-15T15:22:32.290116+0000","last_unstale":"2024-04-15T15:22:35.039242+0000","last_undegraded":"2024-04-15T15:22:35.039242+0000","last_fullsized":"2024-04-15T15:22:35.039242+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:43:31.231190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565305+0000","last_change":"2024-04-15T15:22:33.303094+0000","last_active":"2024-04-15T15:22:34.565305+0000","last_peered":"2024-04-15T15:22:34.565305+0000","last_clean":"2024-04-15T15:22:34.565305+0000","last_became_active":"2024-04-15T15:22:33.302824+0000","last_became_peered":"2024-04-15T15:22:33.302824+0000","last_unstale":"2024-04-15T15:22:34.565305+0000","last_undegraded":"2024-04-15T15:22:34.565305+0000","last_fullsized":"2024-04-15T15:22:34.565305+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:22:11.084708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039278+0000","last_change":"2024-04-15T15:22:32.299094+0000","last_active":"2024-04-15T15:22:35.039278+0000","last_peered":"2024-04-15T15:22:35.039278+0000","last_clean":"2024-04-15T15:22:35.039278+0000","last_became_active":"2024-04-15T15:22:32.298792+0000","last_became_peered":"2024-04-15T15:22:32.298792+0000","last_unstale":"2024-04-15T15:22:35.039278+0000","last_undegraded":"2024-04-15T15:22:35.039278+0000","last_fullsized":"2024-04-15T15:22:35.039278+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:10:15.989931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.305979+0000","last_change":"2024-04-15T15:22:33.296445+0000","last_active":"2024-04-15T15:22:34.305979+0000","last_peered":"2024-04-15T15:22:34.305979+0000","last_clean":"2024-04-15T15:22:34.305979+0000","last_became_active":"2024-04-15T15:22:33.296187+0000","last_became_peered":"2024-04-15T15:22:33.296187+0000","last_unstale":"2024-04-15T15:22:34.305979+0000","last_undegraded":"2024-04-15T15:22:34.305979+0000","last_fullsized":"2024-04-15T15:22:34.305979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:46:17.698031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705414+0000","last_change":"2024-04-15T15:22:32.298790+0000","last_active":"2024-04-15T15:22:34.705414+0000","last_peered":"2024-04-15T15:22:34.705414+0000","last_clean":"2024-04-15T15:22:34.705414+0000","last_became_active":"2024-04-15T15:22:32.298475+0000","last_became_peered":"2024-04-15T15:22:32.298475+0000","last_unstale":"2024-04-15T15:22:34.705414+0000","last_undegraded":"2024-04-15T15:22:34.705414+0000","last_fullsized":"2024-04-15T15:22:34.705414+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:56:45.706167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295502+0000","last_change":"2024-04-15T15:22:33.305871+0000","last_active":"2024-04-15T15:22:34.295502+0000","last_peered":"2024-04-15T15:22:34.295502+0000","last_clean":"2024-04-15T15:22:34.295502+0000","last_became_active":"2024-04-15T15:22:33.305213+0000","last_became_peered":"2024-04-15T15:22:33.305213+0000","last_unstale":"2024-04-15T15:22:34.295502+0000","last_undegraded":"2024-04-15T15:22:34.295502+0000","last_fullsized":"2024-04-15T15:22:34.295502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:18.339496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039592+0000","last_change":"2024-04-15T15:22:32.293012+0000","last_active":"2024-04-15T15:22:35.039592+0000","last_peered":"2024-04-15T15:22:35.039592+0000","last_clean":"2024-04-15T15:22:35.039592+0000","last_became_active":"2024-04-15T15:22:32.292728+0000","last_became_peered":"2024-04-15T15:22:32.292728+0000","last_unstale":"2024-04-15T15:22:35.039592+0000","last_undegraded":"2024-04-15T15:22:35.039592+0000","last_fullsized":"2024-04-15T15:22:35.039592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:17:38.233283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039522+0000","last_change":"2024-04-15T15:22:33.309264+0000","last_active":"2024-04-15T15:22:35.039522+0000","last_peered":"2024-04-15T15:22:35.039522+0000","last_clean":"2024-04-15T15:22:35.039522+0000","last_became_active":"2024-04-15T15:22:33.309005+0000","last_became_peered":"2024-04-15T15:22:33.309005+0000","last_unstale":"2024-04-15T15:22:35.039522+0000","last_undegraded":"2024-04-15T15:22:35.039522+0000","last_fullsized":"2024-04-15T15:22:35.039522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:42:06.582334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306292+0000","last_change":"2024-04-15T15:22:32.295146+0000","last_active":"2024-04-15T15:22:34.306292+0000","last_peered":"2024-04-15T15:22:34.306292+0000","last_clean":"2024-04-15T15:22:34.306292+0000","last_became_active":"2024-04-15T15:22:32.294842+0000","last_became_peered":"2024-04-15T15:22:32.294842+0000","last_unstale":"2024-04-15T15:22:34.306292+0000","last_undegraded":"2024-04-15T15:22:34.306292+0000","last_fullsized":"2024-04-15T15:22:34.306292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:18:34.871504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705452+0000","last_change":"2024-04-15T15:22:33.309039+0000","last_active":"2024-04-15T15:22:34.705452+0000","last_peered":"2024-04-15T15:22:34.705452+0000","last_clean":"2024-04-15T15:22:34.705452+0000","last_became_active":"2024-04-15T15:22:33.308548+0000","last_became_peered":"2024-04-15T15:22:33.308548+0000","last_unstale":"2024-04-15T15:22:34.705452+0000","last_undegraded":"2024-04-15T15:22:34.705452+0000","last_fullsized":"2024-04-15T15:22:34.705452+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:48.797544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039339+0000","last_change":"2024-04-15T15:22:32.295490+0000","last_active":"2024-04-15T15:22:35.039339+0000","last_peered":"2024-04-15T15:22:35.039339+0000","last_clean":"2024-04-15T15:22:35.039339+0000","last_became_active":"2024-04-15T15:22:32.295129+0000","last_became_peered":"2024-04-15T15:22:32.295129+0000","last_unstale":"2024-04-15T15:22:35.039339+0000","last_undegraded":"2024-04-15T15:22:35.039339+0000","last_fullsized":"2024-04-15T15:22:35.039339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:02:52.088900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295379+0000","last_change":"2024-04-15T15:22:33.303705+0000","last_active":"2024-04-15T15:22:34.295379+0000","last_peered":"2024-04-15T15:22:34.295379+0000","last_clean":"2024-04-15T15:22:34.295379+0000","last_became_active":"2024-04-15T15:22:33.303410+0000","last_became_peered":"2024-04-15T15:22:33.303410+0000","last_unstale":"2024-04-15T15:22:34.295379+0000","last_undegraded":"2024-04-15T15:22:34.295379+0000","last_fullsized":"2024-04-15T15:22:34.295379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:33.588528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565454+0000","last_change":"2024-04-15T15:22:32.294246+0000","last_active":"2024-04-15T15:22:34.565454+0000","last_peered":"2024-04-15T15:22:34.565454+0000","last_clean":"2024-04-15T15:22:34.565454+0000","last_became_active":"2024-04-15T15:22:32.293952+0000","last_became_peered":"2024-04-15T15:22:32.293952+0000","last_unstale":"2024-04-15T15:22:34.565454+0000","last_undegraded":"2024-04-15T15:22:34.565454+0000","last_fullsized":"2024-04-15T15:22:34.565454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:36:13.687630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565358+0000","last_change":"2024-04-15T15:22:33.294263+0000","last_active":"2024-04-15T15:22:34.565358+0000","last_peered":"2024-04-15T15:22:34.565358+0000","last_clean":"2024-04-15T15:22:34.565358+0000","last_became_active":"2024-04-15T15:22:33.293971+0000","last_became_peered":"2024-04-15T15:22:33.293971+0000","last_unstale":"2024-04-15T15:22:34.565358+0000","last_undegraded":"2024-04-15T15:22:34.565358+0000","last_fullsized":"2024-04-15T15:22:34.565358+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:49:13.345861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.565507+0000","last_change":"2024-04-15T15:22:32.288288+0000","last_active":"2024-04-15T15:22:34.565507+0000","last_peered":"2024-04-15T15:22:34.565507+0000","last_clean":"2024-04-15T15:22:34.565507+0000","last_became_active":"2024-04-15T15:22:32.287969+0000","last_became_peered":"2024-04-15T15:22:32.287969+0000","last_unstale":"2024-04-15T15:22:34.565507+0000","last_undegraded":"2024-04-15T15:22:34.565507+0000","last_fullsized":"2024-04-15T15:22:34.565507+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:01:27.114232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.039057+0000","last_change":"2024-04-15T15:22:33.300140+0000","last_active":"2024-04-15T15:22:35.039057+0000","last_peered":"2024-04-15T15:22:35.039057+0000","last_clean":"2024-04-15T15:22:35.039057+0000","last_became_active":"2024-04-15T15:22:33.299715+0000","last_became_peered":"2024-04-15T15:22:33.299715+0000","last_unstale":"2024-04-15T15:22:35.039057+0000","last_undegraded":"2024-04-15T15:22:35.039057+0000","last_fullsized":"2024-04-15T15:22:35.039057+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:18:22.291418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:35.038645+0000","last_change":"2024-04-15T15:22:32.292570+0000","last_active":"2024-04-15T15:22:35.038645+0000","last_peered":"2024-04-15T15:22:35.038645+0000","last_clean":"2024-04-15T15:22:35.038645+0000","last_became_active":"2024-04-15T15:22:32.292275+0000","last_became_peered":"2024-04-15T15:22:32.292275+0000","last_unstale":"2024-04-15T15:22:35.038645+0000","last_undegraded":"2024-04-15T15:22:35.038645+0000","last_fullsized":"2024-04-15T15:22:35.038645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:01.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.538124+0000","last_change":"2024-04-15T15:22:33.295370+0000","last_active":"2024-04-15T15:22:34.538124+0000","last_peered":"2024-04-15T15:22:34.538124+0000","last_clean":"2024-04-15T15:22:34.538124+0000","last_became_active":"2024-04-15T15:22:33.295068+0000","last_became_peered":"2024-04-15T15:22:33.295068+0000","last_unstale":"2024-04-15T15:22:34.538124+0000","last_undegraded":"2024-04-15T15:22:34.538124+0000","last_fullsized":"2024-04-15T15:22:34.538124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:14:54.794343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.537995+0000","last_change":"2024-04-15T15:22:32.297615+0000","last_active":"2024-04-15T15:22:34.537995+0000","last_peered":"2024-04-15T15:22:34.537995+0000","last_clean":"2024-04-15T15:22:34.537995+0000","last_became_active":"2024-04-15T15:22:32.297385+0000","last_became_peered":"2024-04-15T15:22:32.297385+0000","last_unstale":"2024-04-15T15:22:34.537995+0000","last_undegraded":"2024-04-15T15:22:34.537995+0000","last_fullsized":"2024-04-15T15:22:34.537995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:33:52.966625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705270+0000","last_change":"2024-04-15T15:22:33.307911+0000","last_active":"2024-04-15T15:22:34.705270+0000","last_peered":"2024-04-15T15:22:34.705270+0000","last_clean":"2024-04-15T15:22:34.705270+0000","last_became_active":"2024-04-15T15:22:33.307610+0000","last_became_peered":"2024-04-15T15:22:33.307610+0000","last_unstale":"2024-04-15T15:22:34.705270+0000","last_undegraded":"2024-04-15T15:22:34.705270+0000","last_fullsized":"2024-04-15T15:22:34.705270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:17:04.067763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.295120+0000","last_change":"2024-04-15T15:22:32.292214+0000","last_active":"2024-04-15T15:22:34.295120+0000","last_peered":"2024-04-15T15:22:34.295120+0000","last_clean":"2024-04-15T15:22:34.295120+0000","last_became_active":"2024-04-15T15:22:32.291892+0000","last_became_peered":"2024-04-15T15:22:32.291892+0000","last_unstale":"2024-04-15T15:22:34.295120+0000","last_undegraded":"2024-04-15T15:22:34.295120+0000","last_fullsized":"2024-04-15T15:22:34.295120+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:47.063192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.306093+0000","last_change":"2024-04-15T15:22:33.301528+0000","last_active":"2024-04-15T15:22:34.306093+0000","last_peered":"2024-04-15T15:22:34.306093+0000","last_clean":"2024-04-15T15:22:34.306093+0000","last_became_active":"2024-04-15T15:22:33.298894+0000","last_became_peered":"2024-04-15T15:22:33.298894+0000","last_unstale":"2024-04-15T15:22:34.306093+0000","last_undegraded":"2024-04-15T15:22:34.306093+0000","last_fullsized":"2024-04-15T15:22:34.306093+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:32.266162+0000","last_clean_scrub_stamp":"2024-04-15T15:22:32.266162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:23:51.004333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705252+0000","last_change":"2024-04-15T15:22:32.299060+0000","last_active":"2024-04-15T15:22:34.705252+0000","last_peered":"2024-04-15T15:22:34.705252+0000","last_clean":"2024-04-15T15:22:34.705252+0000","last_became_active":"2024-04-15T15:22:32.298599+0000","last_became_peered":"2024-04-15T15:22:32.298599+0000","last_unstale":"2024-04-15T15:22:34.705252+0000","last_undegraded":"2024-04-15T15:22:34.705252+0000","last_fullsized":"2024-04-15T15:22:34.705252+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:22:31.260130+0000","last_clean_scrub_stamp":"2024-04-15T15:22:31.260130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:47.063192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:22:34.705488+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:22:34.705488+0000","last_peered":"2024-04-15T15:22:34.705488+0000","last_clean":"2024-04-15T15:22:34.705488+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:22:34.705488+0000","last_undegraded":"2024-04-15T15:22:34.705488+0000","last_fullsized":"2024-04-15T15:22:34.705488+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":1060864,"data_stored":787081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":195613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":458752,"data_stored":195143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":458752,"data_stored":195635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":196677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":195101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":195121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":785433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:22:42.127 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-15T15:22:42.127 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-15T15:22:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.274+0000 7fefae79d640 1 Processor -- start 2024-04-15T15:22:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.274+0000 7fefae79d640 1 -- start start 2024-04-15T15:22:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefae79d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fefa815e790 0x7fefa80d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefae79d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa80d2480 con 0x7fefa80cfd20 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefae79d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa80d2760 con 0x7fefa80d1610 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefae79d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fefa80d2db0 con 0x7fefa815e790 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefa7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fefa815e790 0x7fefa80d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefa7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fefa815e790 0x7fefa80d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49436/0 (socket says 172.21.15.12:49436) 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefa7fff640 1 -- 172.21.15.12:0/3775447070 learned_addr learned my addr 172.21.15.12:0/3775447070 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefa7fff640 1 -- 172.21.15.12:0/3775447070 >> v1:172.21.15.55:6789/0 conn(0x7fefa80d1610 legacy=0x7fefa80b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefa7fff640 1 -- 172.21.15.12:0/3775447070 >> v1:172.21.15.12:6789/0 conn(0x7fefa80cfd20 legacy=0x7fefa80d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.275+0000 7fefa7fff640 1 -- 172.21.15.12:0/3775447070 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa80d2a40 con 0x7fefa815e790 2024-04-15T15:22:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.276+0000 7fefa7fff640 1 --2- 172.21.15.12:0/3775447070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa815e790 0x7fefa80d3240 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fefa80d2f30 tx=0x7fef9802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8890da28998420d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.276+0000 7fefa6ffd640 1 -- 172.21.15.12:0/3775447070 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98034070 con 0x7fefa815e790 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.276+0000 7fefa6ffd640 1 -- 172.21.15.12:0/3775447070 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef98030840 con 0x7fefa815e790 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.276+0000 7fefa6ffd640 1 -- 172.21.15.12:0/3775447070 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98030c10 con 0x7fefa815e790 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.277+0000 7fefae79d640 1 -- 172.21.15.12:0/3775447070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa815e790 msgr2=0x7fefa80d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.277+0000 7fefae79d640 1 --2- 172.21.15.12:0/3775447070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa815e790 0x7fefa80d3240 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fefa80d2f30 tx=0x7fef9802b980 comp rx=0 tx=0).stop 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.277+0000 7fefae79d640 1 -- 172.21.15.12:0/3775447070 shutdown_connections 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.277+0000 7fefae79d640 1 --2- 172.21.15.12:0/3775447070 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa815e790 0x7fefa80d3240 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.277+0000 7fefae79d640 1 -- 172.21.15.12:0/3775447070 >> 172.21.15.12:0/3775447070 conn(0x7fefa808bf60 msgr2=0x7fefa80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.277+0000 7fefae79d640 1 -- 172.21.15.12:0/3775447070 shutdown_connections 2024-04-15T15:22:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.277+0000 7fefae79d640 1 -- 172.21.15.12:0/3775447070 wait complete. 2024-04-15T15:22:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.278+0000 7fefae79d640 1 Processor -- start 2024-04-15T15:22:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.278+0000 7fefae79d640 1 -- start start 2024-04-15T15:22:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefae79d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa80cfd20 0x7fefa815d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefae79d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefa80d1610 0x7fefa81f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefae79d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fefa815e790 0x7fefa81f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefae79d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fefa815ecf0 con 0x7fefa80cfd20 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefae79d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fefa815eb70 con 0x7fefa80d1610 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefae79d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fefa80d0160 con 0x7fefa815e790 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa80cfd20 0x7fefa815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa80cfd20 0x7fefa815d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49442/0 (socket says 172.21.15.12:49442) 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa7fff640 1 -- 172.21.15.12:0/909597492 learned_addr learned my addr 172.21.15.12:0/909597492 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefacd3a640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fefa815e790 0x7fefa81f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa77fe640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefa80d1610 0x7fefa81f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa7fff640 1 -- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fefa815e790 msgr2=0x7fefa81f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa7fff640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fefa815e790 0x7fefa81f47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa7fff640 1 -- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefa80d1610 msgr2=0x7fefa81f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa7fff640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefa80d1610 0x7fefa81f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.279+0000 7fefa7fff640 1 -- 172.21.15.12:0/909597492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa81e4000 con 0x7fefa80cfd20 2024-04-15T15:22:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.280+0000 7fefa7fff640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa80cfd20 0x7fefa815d780 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fef98009870 tx=0x7fef980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.280+0000 7fefa4ff9640 1 -- 172.21.15.12:0/909597492 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98034040 con 0x7fefa80cfd20 2024-04-15T15:22:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.280+0000 7fefa4ff9640 1 -- 172.21.15.12:0/909597492 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef98039d90 con 0x7fefa80cfd20 2024-04-15T15:22:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.280+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa81e76b0 con 0x7fefa80cfd20 2024-04-15T15:22:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.280+0000 7fefa4ff9640 1 -- 172.21.15.12:0/909597492 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98043610 con 0x7fefa80cfd20 2024-04-15T15:22:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.280+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa81e4270 con 0x7fefa80cfd20 2024-04-15T15:22:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.281+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa80689e0 con 0x7fefa80cfd20 2024-04-15T15:22:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.281+0000 7fefa4ff9640 1 -- 172.21.15.12:0/909597492 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef9804a060 con 0x7fefa80cfd20 2024-04-15T15:22:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.282+0000 7fefa4ff9640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef9406c8f0 0x7fef9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.282+0000 7fefa4ff9640 1 -- 172.21.15.12:0/909597492 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fef980adf90 con 0x7fefa80cfd20 2024-04-15T15:22:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.282+0000 7fefa77fe640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef9406c8f0 0x7fef9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.282+0000 7fefa77fe640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef9406c8f0 0x7fef9406edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fefa80a2290 tx=0x7fef9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:42.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.286+0000 7fefa4ff9640 1 -- 172.21.15.12:0/909597492 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef9803e070 con 0x7fefa80cfd20 2024-04-15T15:22:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.544+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fefa80d1ce0 con 0x7fefa80cfd20 2024-04-15T15:22:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.546+0000 7fefa4ff9640 1 -- 172.21.15.12:0/909597492 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7fef9802d330 con 0x7fefa80cfd20 2024-04-15T15:22:42.548 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-04-15T15:22:42.548 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQCkRR1muiw8FBAAiEh+BBQRtVSuiPlO3ZezfA== 2024-04-15T15:22:42.549 INFO:teuthology.orchestra.run.smithi012.stdout: caps mds = "allow" 2024-04-15T15:22:42.549 INFO:teuthology.orchestra.run.smithi012.stdout: caps mon = "allow r" 2024-04-15T15:22:42.549 INFO:teuthology.orchestra.run.smithi012.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-15T15:22:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:updated caps for client.0 2024-04-15T15:22:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef9406c8f0 msgr2=0x7fef9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef9406c8f0 0x7fef9406edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fefa80a2290 tx=0x7fef9c009040 comp rx=0 tx=0).stop 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa80cfd20 msgr2=0x7fefa815d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa80cfd20 0x7fefa815d780 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fef98009870 tx=0x7fef980046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 shutdown_connections 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fefa815e790 0x7fefa81f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fef9406c8f0 0x7fef9406edb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefa80d1610 0x7fefa81f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 --2- 172.21.15.12:0/909597492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefa80cfd20 0x7fefa815d780 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 >> 172.21.15.12:0/909597492 conn(0x7fefa808bf60 msgr2=0x7fefa80b8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 shutdown_connections 2024-04-15T15:22:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.549+0000 7fefae79d640 1 -- 172.21.15.12:0/909597492 wait complete. 2024-04-15T15:22:42.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:22:42.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.665+0000 7fa651ae0640 1 Processor -- start 2024-04-15T15:22:42.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.665+0000 7fa651ae0640 1 -- start start 2024-04-15T15:22:42.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa651ae0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa64c15e790 0x7fa64c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa651ae0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa64c0d1cf0 0x7fa64c0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa651ae0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa64c15eb70 con 0x7fa64c0b7be0 2024-04-15T15:22:42.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa651ae0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa64c0d3790 con 0x7fa64c15e790 2024-04-15T15:22:42.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa651ae0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa64c0d45b0 con 0x7fa64c0d1cf0 2024-04-15T15:22:42.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa64b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa64c15e790 0x7fa64c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa64b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa64c15e790 0x7fa64c0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49448/0 (socket says 172.21.15.12:49448) 2024-04-15T15:22:42.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa64b7fe640 1 -- 172.21.15.12:0/1258240963 learned_addr learned my addr 172.21.15.12:0/1258240963 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:42.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa64bfff640 1 --2- 172.21.15.12:0/1258240963 >> v2:172.21.15.55:3300/0 conn(0x7fa64c0d1cf0 0x7fa64c0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.666+0000 7fa64b7fe640 1 -- 172.21.15.12:0/1258240963 >> v2:172.21.15.55:3300/0 conn(0x7fa64c0d1cf0 msgr2=0x7fa64c0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:42.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.667+0000 7fa64b7fe640 1 --2- 172.21.15.12:0/1258240963 >> v2:172.21.15.55:3300/0 conn(0x7fa64c0d1cf0 0x7fa64c0d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.667+0000 7fa64b7fe640 1 -- 172.21.15.12:0/1258240963 >> v1:172.21.15.55:6789/0 conn(0x7fa64c0b7be0 legacy=0x7fa64c0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.667+0000 7fa64b7fe640 1 -- 172.21.15.12:0/1258240963 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64c0d3c80 con 0x7fa64c15e790 2024-04-15T15:22:42.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.668+0000 7fa64b7fe640 1 --2- 172.21.15.12:0/1258240963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0b76a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0b61c0 tx=0x7fa63c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77462b706284e29c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:42.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.669+0000 7fa64affd640 1 -- 172.21.15.12:0/1258240963 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c02e070 con 0x7fa64c15e790 2024-04-15T15:22:42.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.669+0000 7fa64affd640 1 -- 172.21.15.12:0/1258240963 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa63c037070 con 0x7fa64c15e790 2024-04-15T15:22:42.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.669+0000 7fa64affd640 1 -- 172.21.15.12:0/1258240963 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c031a00 con 0x7fa64c15e790 2024-04-15T15:22:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.669+0000 7fa651ae0640 1 -- 172.21.15.12:0/1258240963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 msgr2=0x7fa64c0b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.669+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1258240963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0b76a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0b61c0 tx=0x7fa63c02b880 comp rx=0 tx=0).stop 2024-04-15T15:22:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.670+0000 7fa651ae0640 1 -- 172.21.15.12:0/1258240963 shutdown_connections 2024-04-15T15:22:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.670+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1258240963 >> v2:172.21.15.55:3300/0 conn(0x7fa64c0d1cf0 0x7fa64c0d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.670+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1258240963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0b76a0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.670+0000 7fa651ae0640 1 -- 172.21.15.12:0/1258240963 >> 172.21.15.12:0/1258240963 conn(0x7fa64c067b40 msgr2=0x7fa64c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.670+0000 7fa651ae0640 1 -- 172.21.15.12:0/1258240963 shutdown_connections 2024-04-15T15:22:42.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.670+0000 7fa651ae0640 1 -- 172.21.15.12:0/1258240963 wait complete. 2024-04-15T15:22:42.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.671+0000 7fa651ae0640 1 Processor -- start 2024-04-15T15:22:42.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.671+0000 7fa651ae0640 1 -- start start 2024-04-15T15:22:42.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa651ae0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa64c0b7be0 0x7fa64c0cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa651ae0640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa64c0d1cf0 0x7fa64c0b9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa651ae0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa651ae0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c0d4f80 con 0x7fa64c15e790 2024-04-15T15:22:42.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa651ae0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c0d4240 con 0x7fa64c0b7be0 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa651ae0640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa64c0d5100 con 0x7fa64c0d1cf0 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0c6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49456/0 (socket says 172.21.15.12:49456) 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 -- 172.21.15.12:0/1531095610 learned_addr learned my addr 172.21.15.12:0/1531095610 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64b7fe640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa64c0b7be0 0x7fa64c0cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa642ffd640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa64c0d1cf0 0x7fa64c0b9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 -- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa64c0d1cf0 msgr2=0x7fa64c0b9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa64c0d1cf0 0x7fa64c0b9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 -- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa64c0b7be0 msgr2=0x7fa64c0cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa64c0b7be0 0x7fa64c0cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 -- 172.21.15.12:0/1531095610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64c0cc2f0 con 0x7fa64c15e790 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa64bfff640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0c6d30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa644009d20 tx=0x7fa64400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa648ff9640 1 -- 172.21.15.12:0/1531095610 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644018020 con 0x7fa64c15e790 2024-04-15T15:22:42.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa648ff9640 1 -- 172.21.15.12:0/1531095610 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa644009b10 con 0x7fa64c15e790 2024-04-15T15:22:42.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64c0cc5d0 con 0x7fa64c15e790 2024-04-15T15:22:42.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.672+0000 7fa648ff9640 1 -- 172.21.15.12:0/1531095610 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6440146c0 con 0x7fa64c15e790 2024-04-15T15:22:42.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.673+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64c0ccb90 con 0x7fa64c15e790 2024-04-15T15:22:42.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.673+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa64c074be0 con 0x7fa64c15e790 2024-04-15T15:22:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.674+0000 7fa648ff9640 1 -- 172.21.15.12:0/1531095610 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa644005d10 con 0x7fa64c15e790 2024-04-15T15:22:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.674+0000 7fa648ff9640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa62c06c8f0 0x7fa62c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.674+0000 7fa648ff9640 1 -- 172.21.15.12:0/1531095610 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa64408c6c0 con 0x7fa64c15e790 2024-04-15T15:22:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.674+0000 7fa64b7fe640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa62c06c8f0 0x7fa62c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:42.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.675+0000 7fa64b7fe640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa62c06c8f0 0x7fa62c06edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa63c036000 tx=0x7fa63c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:42.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.679+0000 7fa648ff9640 1 -- 172.21.15.12:0/1531095610 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa64c074be0 con 0x7fa64c15e790 2024-04-15T15:22:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.900+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa64c0689e0 con 0x7fa64c15e790 2024-04-15T15:22:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.900+0000 7fa648ff9640 1 -- 172.21.15.12:0/1531095610 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7fa644022070 con 0x7fa64c15e790 2024-04-15T15:22:42.903 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:42.903 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:22:35.667510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/3388147599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":3388147599},{"type":"v1","addr":"172.21.15.55:6837","nonce":3388147599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5203,"qdb_cluster":[5203]},"id":2}]} 2024-04-15T15:22:42.904 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 18 2024-04-15T15:22:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa62c06c8f0 msgr2=0x7fa62c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa62c06c8f0 0x7fa62c06edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa63c036000 tx=0x7fa63c03d040 comp rx=0 tx=0).stop 2024-04-15T15:22:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 msgr2=0x7fa64c0c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0c6d30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa644009d20 tx=0x7fa64400c490 comp rx=0 tx=0).stop 2024-04-15T15:22:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 shutdown_connections 2024-04-15T15:22:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa64c15e790 0x7fa64c0c6d30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa64c0d1cf0 0x7fa64c0b9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa62c06c8f0 0x7fa62c06edb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 --2- 172.21.15.12:0/1531095610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa64c0b7be0 0x7fa64c0cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.902+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 >> 172.21.15.12:0/1531095610 conn(0x7fa64c067b40 msgr2=0x7fa64c0cf4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.903+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 shutdown_connections 2024-04-15T15:22:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:42.903+0000 7fa651ae0640 1 -- 172.21.15.12:0/1531095610 wait complete. 2024-04-15T15:22:43.319 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:22:32.656644+0000', 'modified': '2024-04-15T15:22:35.667510+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5203}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5203': {'gid': 5203, 'name': 'd', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.55:6837/3388147599', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6835', 'nonce': 3388147599}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 3388147599}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5203, 'qdb_cluster': [5203]} 2024-04-15T15:22:43.319 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:22:43.319 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-15T15:22:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.427+0000 7fc519e63640 1 Processor -- start 2024-04-15T15:22:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.427+0000 7fc519e63640 1 -- start start 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc519e63640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc5140d09b0 0x7fc51409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc519e63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 0x7fc514160770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc519e63640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5140d1390 con 0x7fc514160390 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc519e63640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc5140cd790 con 0x7fc5140d09b0 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc519e63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc5140ce330 con 0x7fc51409af10 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc5137fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc514160390 0x7fc5140cdc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42724/0 (socket says 172.21.15.12:42724) 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc5137fe640 1 -- 172.21.15.12:0/3880740797 learned_addr learned my addr 172.21.15.12:0/3880740797 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc513fff640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 0x7fc514160770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc512ffd640 1 --2- 172.21.15.12:0/3880740797 >> v2:172.21.15.55:3300/0 conn(0x7fc5140d09b0 0x7fc51409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497894269 0 0) 0x7fc5140d1390 con 0x7fc514160390 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.428+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc500003520 con 0x7fc514160390 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2586848101 0 0) 0x7fc500003520 con 0x7fc514160390 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 msgr2=0x7fc514160770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 0x7fc514160770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 >> v2:172.21.15.55:3300/0 conn(0x7fc5140d09b0 msgr2=0x7fc51409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 --2- 172.21.15.12:0/3880740797 >> v2:172.21.15.55:3300/0 conn(0x7fc5140d09b0 0x7fc51409a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5140ccea0 con 0x7fc514160390 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fc504001670 con 0x7fc514160390 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc513fff640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 0x7fc514160770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc512ffd640 1 --2- 172.21.15.12:0/3880740797 >> v2:172.21.15.55:3300/0 conn(0x7fc5140d09b0 0x7fc51409a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 >> v1:172.21.15.12:6789/0 conn(0x7fc514160390 legacy=0x7fc5140cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5000039b0 0x7fc500003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 0x7fc5000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc5140cd790 con 0x7fc5000039b0 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc5140ce330 con 0x7fc500004800 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc50000b9c0 con 0x7fc51409af10 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5137fe640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5000039b0 0x7fc500003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc512ffd640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc513fff640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 0x7fc5000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5137fe640 1 -- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 msgr2=0x7fc5000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5137fe640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc51409af10 0x7fc5000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5137fe640 1 -- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc500004800 msgr2=0x7fc500004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5137fe640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.429+0000 7fc5137fe640 1 -- 172.21.15.12:0/3880740797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc500003520 con 0x7fc5000039b0 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.430+0000 7fc5137fe640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5000039b0 0x7fc500003dd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc50400c210 tx=0x7fc500005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5d5290683b2640 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.430+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504007d50 con 0x7fc5000039b0 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.430+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc504030070 con 0x7fc5000039b0 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.430+0000 7fc5127fc640 1 -- 172.21.15.12:0/3880740797 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50402cd80 con 0x7fc5000039b0 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.430+0000 7fc519e63640 1 -- 172.21.15.12:0/3880740797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5000039b0 msgr2=0x7fc500003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.430+0000 7fc519e63640 1 --2- 172.21.15.12:0/3880740797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5000039b0 0x7fc500003dd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc50400c210 tx=0x7fc500005830 comp rx=0 tx=0).stop 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.431+0000 7fc513fff640 1 -- 172.21.15.12:0/3880740797 reap_dead start 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.431+0000 7fc519e63640 1 -- 172.21.15.12:0/3880740797 shutdown_connections 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.431+0000 7fc519e63640 1 -- 172.21.15.12:0/3880740797 >> 172.21.15.12:0/3880740797 conn(0x7fc5140b3ec0 msgr2=0x7fc5140b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.431+0000 7fc519e63640 1 -- 172.21.15.12:0/3880740797 shutdown_connections 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.431+0000 7fc519e63640 1 -- 172.21.15.12:0/3880740797 wait complete. 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.432+0000 7fc519e63640 1 Processor -- start 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.432+0000 7fc519e63640 1 -- start start 2024-04-15T15:22:43.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc519e63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5000039b0 0x7fc5141f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc519e63640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5140b1330 0x7fc514158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc519e63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5140c94a0 0x7fc5141f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc519e63640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc5140cd140 con 0x7fc5140c94a0 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc519e63640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc5140d0e10 con 0x7fc5140b1330 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc519e63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc5140cd420 con 0x7fc5000039b0 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5140c94a0 0x7fc5141f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5140c94a0 0x7fc5141f76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49482/0 (socket says 172.21.15.12:49482) 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 -- 172.21.15.12:0/392795980 learned_addr learned my addr 172.21.15.12:0/392795980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc512ffd640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5140b1330 0x7fc514158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc5137fe640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5000039b0 0x7fc5141f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 -- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5000039b0 msgr2=0x7fc5141f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5000039b0 0x7fc5141f4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 -- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5140b1330 msgr2=0x7fc514158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5140b1330 0x7fc514158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 -- 172.21.15.12:0/392795980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5140c9e10 con 0x7fc5140c94a0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc512ffd640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5140b1330 0x7fc514158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc513fff640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5140c94a0 0x7fc5141f76a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc514073df0 tx=0x7fc50c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc50bfff640 1 -- 172.21.15.12:0/392795980 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c011dd0 con 0x7fc5140c94a0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc50bfff640 1 -- 172.21.15.12:0/392795980 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc50c01b070 con 0x7fc5140c94a0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc50bfff640 1 -- 172.21.15.12:0/392795980 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c003990 con 0x7fc5140c94a0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc519e63640 1 -- 172.21.15.12:0/392795980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5140ca0f0 con 0x7fc5140c94a0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.433+0000 7fc519e63640 1 -- 172.21.15.12:0/392795980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5140ca6b0 con 0x7fc5140c94a0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.435+0000 7fc50bfff640 1 -- 172.21.15.12:0/392795980 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc50c003b30 con 0x7fc5140c94a0 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.435+0000 7fc50bfff640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4f406c8f0 0x7fc4f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.436+0000 7fc50bfff640 1 -- 172.21.15.12:0/392795980 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc50c08c4f0 con 0x7fc5140c94a0 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.436+0000 7fc5137fe640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4f406c8f0 0x7fc4f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.436+0000 7fc5137fe640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4f406c8f0 0x7fc4f406edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc504034000 tx=0x7fc504007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.436+0000 7fc51ca4e740 1 Processor -- start 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.436+0000 7fc51ca4e740 1 -- start start 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35c22f0 0x5607a35f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5607a352c9c0 con 0x5607a35ef1f0 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5607a34d3650 con 0x5607a35eee10 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x5607a3528980 con 0x5607a35c22f0 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc512ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc512ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49486/0 (socket says 172.21.15.12:49486) 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.437+0000 7fc512ffd640 1 -- 172.21.15.12:0/4009672247 learned_addr learned my addr 172.21.15.12:0/4009672247 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:43.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc513fff640 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35c22f0 0x5607a35f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc5137fe640 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc512ffd640 1 -- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35c22f0 msgr2=0x5607a35f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc512ffd640 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35c22f0 0x5607a35f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc512ffd640 1 -- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 msgr2=0x5607a35c1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc512ffd640 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc512ffd640 1 -- 172.21.15.12:0/4009672247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607a323f2d0 con 0x5607a35ef1f0 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc5137fe640 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc512ffd640 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35f1ba0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009c40 tx=0x7fc4fc009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3567eb7e9d8c8825 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc5097fa640 1 -- 172.21.15.12:0/4009672247 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc01b070 con 0x5607a35ef1f0 2024-04-15T15:22:43.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc5097fa640 1 -- 172.21.15.12:0/4009672247 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc013e70 con 0x5607a35ef1f0 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.438+0000 7fc5097fa640 1 -- 172.21.15.12:0/4009672247 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc016660 con 0x5607a35ef1f0 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 -- 172.21.15.12:0/4009672247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 msgr2=0x5607a35f1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35f1ba0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009c40 tx=0x7fc4fc009190 comp rx=0 tx=0).stop 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 -- 172.21.15.12:0/4009672247 shutdown_connections 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35c22f0 0x5607a35f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35f1ba0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/4009672247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 -- 172.21.15.12:0/4009672247 >> 172.21.15.12:0/4009672247 conn(0x5607a35c4020 msgr2=0x5607a360cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 -- 172.21.15.12:0/4009672247 shutdown_connections 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 -- 172.21.15.12:0/4009672247 wait complete. 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.439+0000 7fc51ca4e740 1 -- start start 2024-04-15T15:22:43.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc51ca4e740 1 Processor -- start 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35ef1f0 0x5607a3607780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35dc930 0x5607a361ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5607a35c65c0 con 0x5607a35eee10 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5607a34e6a10 con 0x5607a35dc930 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x5607a35c59b0 con 0x5607a35ef1f0 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc51ca4e740 10 client.0 _ll_register_callbacks cb 0x5607a3522250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc5137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc5137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49488/0 (socket says 172.21.15.12:49488) 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc5137fe640 1 -- 172.21.15.12:0/1673446635 learned_addr learned my addr 172.21.15.12:0/1673446635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.440+0000 7fc512ffd640 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35ef1f0 0x5607a3607780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc513fff640 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35dc930 0x5607a361ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc5137fe640 1 -- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35ef1f0 msgr2=0x5607a3607780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc5137fe640 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35ef1f0 0x5607a3607780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc5137fe640 1 -- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35dc930 msgr2=0x5607a361ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc5137fe640 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35dc930 0x5607a361ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc5137fe640 1 -- 172.21.15.12:0/1673446635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607a33096b0 con 0x5607a35eee10 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc513fff640 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35dc930 0x5607a361ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc5137fe640 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f48a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc50400c300 tx=0x7fc504001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d0dfd004eaebdd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc4f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1673446635 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504052060 con 0x5607a35eee10 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1673446635 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc504007480 con 0x5607a35eee10 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1673446635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4fc013e70 con 0x5607a35eee10 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1673446635 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fc504007680 con 0x5607a35eee10 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc51ca4e740 10 client.5694 fetch_fsmap learned FSMap version 18 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1673446635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x5607a35deb40 con 0x5607a35eee10 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1673446635 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc013e70 con 0x5607a35eee10 2024-04-15T15:22:43.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.441+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1673446635 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7fc50404c030 con 0x5607a35eee10 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc51ca4e740 10 client.5694 fetch_fsmap finished waiting for FSMap version 18 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc51ca4e740 10 client.5694 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc51ca4e740 20 client.5694 populate_metadata read hostname 'smithi012' 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x5607a3606ae0 0x5607a35dfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc51ca4e740 4 client.5694 mds_command: new command op to 5203 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1673446635 --> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607a353a3e0 con 0x5607a3606ae0 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc512ffd640 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x5607a3606ae0 0x5607a35dfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc4f27fc640 10 client.5694 ms_handle_connect on v2:172.21.15.55:6835/3388147599 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.442+0000 7fc512ffd640 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x5607a3606ae0 0x5607a35dfdd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46b2064f08b18ef3 server_cookie=566060d8c96c68ed in_seq=0 out_seq=0 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1673446635 <== mds.0 v2:172.21.15.55:6835/3388147599 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5607a353a3e0 con 0x5607a3606ae0 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc4f27fc640 10 client.5694 handle_command_reply: tid=0 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 client.5694 shutdown 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1673446635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 msgr2=0x5607a35f48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f48a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc50400c300 tx=0x7fc504001ab0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1673446635 shutdown_connections 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x5607a3606ae0 0x5607a35dfdd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35dc930 0x5607a361ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc4f27fc640 0 client.5694 ms_handle_reset on v2:172.21.15.55:6835/3388147599 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35ef1f0 0x5607a3607780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1673446635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f48a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1673446635 >> 172.21.15.12:0/1673446635 conn(0x5607a35c22f0 msgr2=0x5607a35f1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1673446635 shutdown_connections 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1673446635 wait complete. 2024-04-15T15:22:43.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.443+0000 7fc51ca4e740 20 client.5694 trim_cache size 0 max 16384 2024-04-15T15:22:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.461+0000 7fc51ca4e740 1 Processor -- start 2024-04-15T15:22:43.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.461+0000 7fc51ca4e740 1 -- start start 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 0x5607a360f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35ef1f0 0x5607a3610490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc514200a60 con 0x5607a35eee10 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc5140cb760 con 0x5607a35ef1f0 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc514200510 con 0x5607a35dc930 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc512ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc512ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49502/0 (socket says 172.21.15.12:49502) 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.462+0000 7fc512ffd640 1 -- 172.21.15.12:0/487262693 learned_addr learned my addr 172.21.15.12:0/487262693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:43.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc512ffd640 1 -- 172.21.15.12:0/487262693 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 msgr2=0x5607a360f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc512ffd640 1 --2- 172.21.15.12:0/487262693 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 0x5607a360f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc512ffd640 1 -- 172.21.15.12:0/487262693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35ef1f0 msgr2=0x5607a3610490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc512ffd640 1 --2- 172.21.15.12:0/487262693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35ef1f0 0x5607a3610490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc512ffd640 1 -- 172.21.15.12:0/487262693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607a35f58c0 con 0x5607a35eee10 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc512ffd640 1 --2- 172.21.15.12:0/487262693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f1d80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc013b40 tx=0x7fc4fc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d6a3d6c5e2c99d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc508ff9640 1 -- 172.21.15.12:0/487262693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc01b040 con 0x5607a35eee10 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc508ff9640 1 -- 172.21.15.12:0/487262693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc02f2e0 con 0x5607a35eee10 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc508ff9640 1 -- 172.21.15.12:0/487262693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc02f6d0 con 0x5607a35eee10 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc51ca4e740 1 -- 172.21.15.12:0/487262693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 msgr2=0x5607a35f1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.463+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/487262693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f1d80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc013b40 tx=0x7fc4fc018000 comp rx=0 tx=0).stop 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 -- 172.21.15.12:0/487262693 shutdown_connections 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/487262693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35ef1f0 0x5607a3610490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/487262693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35eee10 0x5607a35f1d80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/487262693 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 0x5607a360f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 -- 172.21.15.12:0/487262693 >> 172.21.15.12:0/487262693 conn(0x5607a35c22f0 msgr2=0x5607a35c7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:43.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 -- 172.21.15.12:0/487262693 shutdown_connections 2024-04-15T15:22:43.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 -- 172.21.15.12:0/487262693 wait complete. 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 -- start start 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.464+0000 7fc51ca4e740 1 Processor -- start 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 0x5607a3686ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc51ca4e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5607a3606960 con 0x5607a35ef1f0 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5607a3606510 con 0x5607a35eee10 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc51ca4e740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x5607a3606f80 con 0x5607a35dc930 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc51ca4e740 10 client.0 _ll_register_callbacks cb 0x7fc5000036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35c57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49510/0 (socket says 172.21.15.12:49510) 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 -- 172.21.15.12:0/1201323945 learned_addr learned my addr 172.21.15.12:0/1201323945 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc512ffd640 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc5137fe640 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 0x5607a3686ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 -- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 msgr2=0x5607a3686ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 0x5607a3686ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 -- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 msgr2=0x5607a35c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 -- 172.21.15.12:0/1201323945 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607a3610c30 con 0x5607a35ef1f0 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc5137fe640 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 0x5607a3686ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc513fff640 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35c57e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0627d0 tx=0x7fc50c05a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2645473683b4ea8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.465+0000 7fc4f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:22:43.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1201323945 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c05e2c0 con 0x5607a35ef1f0 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1201323945 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc50c08c0c0 con 0x5607a35ef1f0 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1201323945 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4fc02f2e0 con 0x5607a35ef1f0 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1201323945 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fc50c062d00 con 0x5607a35ef1f0 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 10 client.5700 fetch_fsmap learned FSMap version 18 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1201323945 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x5607a3608080 con 0x5607a35ef1f0 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1201323945 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc02f2e0 con 0x5607a35ef1f0 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1201323945 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7fc50c062870 con 0x5607a35ef1f0 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 10 client.5700 fetch_fsmap finished waiting for FSMap version 18 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 10 client.5700 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:22:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 20 client.5700 populate_metadata read hostname 'smithi012' 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x5607a3606ae0 0x5607a3687400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 4 client.5700 mds_command: new command op to 5203 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1201323945 --> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] -- command(tid 0: {"prefix": "status"}) -- 0x5607a3534250 con 0x5607a3606ae0 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.466+0000 7fc5137fe640 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x5607a3606ae0 0x5607a3687400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc4f27fc640 10 client.5700 ms_handle_connect on v2:172.21.15.55:6835/3388147599 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc5137fe640 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x5607a3606ae0 0x5607a3687400 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71e09768d8683c68 server_cookie=569678b387441dcd in_seq=0 out_seq=0 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc4f27fc640 1 -- 172.21.15.12:0/1201323945 <== mds.0 v2:172.21.15.55:6835/3388147599 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5607a3534250 con 0x5607a3606ae0 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc4f27fc640 10 client.5700 handle_command_reply: tid=0 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 client.5700 shutdown 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1201323945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 msgr2=0x5607a35c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35c57e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0627d0 tx=0x7fc50c05a1e0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1201323945 shutdown_connections 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607a35ef1f0 0x5607a35c57e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x5607a3606ae0 0x5607a3687400 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5607a35eee10 0x5607a35c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc4f27fc640 0 client.5700 ms_handle_reset on v2:172.21.15.55:6835/3388147599 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 --2- 172.21.15.12:0/1201323945 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x5607a35dc930 0x5607a3686ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1201323945 >> 172.21.15.12:0/1201323945 conn(0x5607a35c22f0 msgr2=0x5607a36075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1201323945 shutdown_connections 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 1 -- 172.21.15.12:0/1201323945 wait complete. 2024-04-15T15:22:43.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.467+0000 7fc51ca4e740 20 client.5700 trim_cache size 0 max 16384 2024-04-15T15:22:43.478 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:22:43.478 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "928fa63b-2322-46eb-a6b8-ef2b880cd7c9", 2024-04-15T15:22:43.478 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-04-15T15:22:43.478 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5203, 2024-04-15T15:22:43.478 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-15T15:22:43.478 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-15T15:22:43.478 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-15T15:22:43.479 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 10.177661403, 2024-04-15T15:22:43.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 18, 2024-04-15T15:22:43.479 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 39, 2024-04-15T15:22:43.479 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 39, 2024-04-15T15:22:43.479 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 17.273014590999999 2024-04-15T15:22:43.479 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:22:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 -- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4f406c8f0 msgr2=0x7fc4f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4f406c8f0 0x7fc4f406edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc504034000 tx=0x7fc504007000 comp rx=0 tx=0).stop 2024-04-15T15:22:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 -- 172.21.15.12:0/392795980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5140c94a0 msgr2=0x7fc5141f76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5140c94a0 0x7fc5141f76a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc514073df0 tx=0x7fc50c009f90 comp rx=0 tx=0).stop 2024-04-15T15:22:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 -- 172.21.15.12:0/392795980 shutdown_connections 2024-04-15T15:22:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5140c94a0 0x7fc5141f76a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4f406c8f0 0x7fc4f406edb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5140b1330 0x7fc514158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 --2- 172.21.15.12:0/392795980 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc5000039b0 0x7fc5141f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 -- 172.21.15.12:0/392795980 >> 172.21.15.12:0/392795980 conn(0x7fc5140b3ec0 msgr2=0x7fc5140b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:43.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 -- 172.21.15.12:0/392795980 shutdown_connections 2024-04-15T15:22:43.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.468+0000 7fc519e63640 1 -- 172.21.15.12:0/392795980 wait complete. 2024-04-15T15:22:43.482 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:22:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.585+0000 7f1e3e40c640 1 Processor -- start 2024-04-15T15:22:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.585+0000 7f1e3e40c640 1 -- start start 2024-04-15T15:22:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e3e40c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e3e40c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1e380ae7d0 0x7f1e38071e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e3e40c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e380aee30 con 0x7f1e3806fbc0 2024-04-15T15:22:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e3e40c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1e38070000 con 0x7f1e380ae7d0 2024-04-15T15:22:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e3e40c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1e380af110 con 0x7f1e38158de0 2024-04-15T15:22:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e37fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e3c9a9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1e380ae7d0 0x7f1e38071e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e2ffff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1e3806fbc0 0x7f1e380ae0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:36362/0 (socket says 172.21.15.12:36362) 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e3c9a9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1e380ae7d0 0x7f1e38071e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:43126/0 (socket says 172.21.15.12:43126) 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e37fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:55520/0 (socket says 172.21.15.12:55520) 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.586+0000 7f1e2ffff640 1 -- 172.21.15.12:0/948250488 learned_addr learned my addr 172.21.15.12:0/948250488 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.587+0000 7f1e3c9a9640 1 -- 172.21.15.12:0/948250488 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 msgr2=0x7f1e3806f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.587+0000 7f1e3c9a9640 1 --2- 172.21.15.12:0/948250488 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.587+0000 7f1e3c9a9640 1 -- 172.21.15.12:0/948250488 >> v1:172.21.15.55:6789/0 conn(0x7f1e3806fbc0 legacy=0x7f1e380ae0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.587+0000 7f1e3c9a9640 1 -- 172.21.15.12:0/948250488 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e380af3f0 con 0x7f1e380ae7d0 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.587+0000 7f1e377fe640 1 -- 172.21.15.12:0/948250488 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3430097040 0 0) 0x7f1e380aee30 con 0x7f1e3806fbc0 2024-04-15T15:22:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.587+0000 7f1e37fff640 1 --2- 172.21.15.12:0/948250488 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3c9a9640 1 --2- 172.21.15.12:0/948250488 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 0x7f1e38071e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1e30009d90 tx=0x7f1e3002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a4b9a36a9c6d0a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e377fe640 1 -- 172.21.15.12:0/948250488 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30039070 con 0x7f1e380ae7d0 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e377fe640 1 -- 172.21.15.12:0/948250488 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3002cc50 con 0x7f1e380ae7d0 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e377fe640 1 -- 172.21.15.12:0/948250488 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30034440 con 0x7f1e380ae7d0 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3e40c640 1 -- 172.21.15.12:0/948250488 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 msgr2=0x7f1e38071e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/948250488 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 0x7f1e38071e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1e30009d90 tx=0x7f1e3002bea0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3e40c640 1 -- 172.21.15.12:0/948250488 shutdown_connections 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/948250488 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 0x7f1e38071e30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/948250488 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3e40c640 1 -- 172.21.15.12:0/948250488 >> 172.21.15.12:0/948250488 conn(0x7f1e380b1f60 msgr2=0x7f1e380b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3e40c640 1 -- 172.21.15.12:0/948250488 shutdown_connections 2024-04-15T15:22:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.588+0000 7f1e3e40c640 1 -- 172.21.15.12:0/948250488 wait complete. 2024-04-15T15:22:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e3e40c640 1 Processor -- start 2024-04-15T15:22:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e3e40c640 1 -- start start 2024-04-15T15:22:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e3e40c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3806fbc0 0x7f1e3814e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e3e40c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 0x7f1e3806e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e3e40c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e3e40c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1e381591c0 con 0x7f1e3806fbc0 2024-04-15T15:22:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e3e40c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1e380af9b0 con 0x7f1e380ae7d0 2024-04-15T15:22:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e3e40c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1e38159340 con 0x7f1e38158de0 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e37fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3806fbc0 0x7f1e3814e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e37fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3806fbc0 0x7f1e3814e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49518/0 (socket says 172.21.15.12:49518) 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.590+0000 7f1e37fff640 1 -- 172.21.15.12:0/2138092373 learned_addr learned my addr 172.21.15.12:0/2138092373 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e3c9a9640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e2ffff640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 0x7f1e3806e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e37fff640 1 -- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 msgr2=0x7f1e3806eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e37fff640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e37fff640 1 -- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 msgr2=0x7f1e3806e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e37fff640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 0x7f1e3806e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e37fff640 1 -- 172.21.15.12:0/2138092373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e380b4480 con 0x7f1e3806fbc0 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e2ffff640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 0x7f1e3806e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e37fff640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3806fbc0 0x7f1e3814e810 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800ebe0 tx=0x7f1e2800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e357fa640 1 -- 172.21.15.12:0/2138092373 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28017070 con 0x7f1e3806fbc0 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e357fa640 1 -- 172.21.15.12:0/2138092373 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e2800cc30 con 0x7f1e3806fbc0 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e380b4760 con 0x7f1e3806fbc0 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e357fa640 1 -- 172.21.15.12:0/2138092373 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28013470 con 0x7f1e3806fbc0 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.591+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e380b4d20 con 0x7f1e3806fbc0 2024-04-15T15:22:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.592+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e380689e0 con 0x7f1e3806fbc0 2024-04-15T15:22:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.593+0000 7f1e357fa640 1 -- 172.21.15.12:0/2138092373 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e2801a070 con 0x7f1e3806fbc0 2024-04-15T15:22:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.593+0000 7f1e357fa640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e1806c8f0 0x7f1e1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.594+0000 7f1e357fa640 1 -- 172.21.15.12:0/2138092373 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f1e2808bfd0 con 0x7f1e3806fbc0 2024-04-15T15:22:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.594+0000 7f1e2ffff640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e1806c8f0 0x7f1e1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.594+0000 7f1e2ffff640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e1806c8f0 0x7f1e1806edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1e3806f3c0 tx=0x7f1e20009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.598+0000 7f1e357fa640 1 -- 172.21.15.12:0/2138092373 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e2805a270 con 0x7f1e3806fbc0 2024-04-15T15:22:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.824+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1e3806e060 con 0x7f1e3806fbc0 2024-04-15T15:22:43.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.825+0000 7f1e357fa640 1 -- 172.21.15.12:0/2138092373 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7f1e28059b70 con 0x7f1e3806fbc0 2024-04-15T15:22:43.827 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:22:43.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e1806c8f0 msgr2=0x7f1e1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e1806c8f0 0x7f1e1806edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1e3806f3c0 tx=0x7f1e20009040 comp rx=0 tx=0).stop 2024-04-15T15:22:43.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3806fbc0 msgr2=0x7f1e3814e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:43.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3806fbc0 0x7f1e3814e810 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800ebe0 tx=0x7f1e2800c520 comp rx=0 tx=0).stop 2024-04-15T15:22:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 shutdown_connections 2024-04-15T15:22:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1e38158de0 0x7f1e3806eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1e1806c8f0 0x7f1e1806edb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e380ae7d0 0x7f1e3806e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 --2- 172.21.15.12:0/2138092373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3806fbc0 0x7f1e3814e810 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 >> 172.21.15.12:0/2138092373 conn(0x7f1e380b1f60 msgr2=0x7f1e38071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 shutdown_connections 2024-04-15T15:22:43.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:43.827+0000 7f1e3e40c640 1 -- 172.21.15.12:0/2138092373 wait complete. 2024-04-15T15:22:43.840 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:22:43.840 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:22:43.840 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-04-15T15:22:43.840 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:22:43.841 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:22:43.841 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:22:43.841 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:22:43.841 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:22:43.841 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:22:43.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:43.841 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-04-15T15:22:43.856 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:43.857 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42585sec preferred_lft 42585sec 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 36:f2:47:76:c7:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::18d2:66ff:fef6:52a/64 scope link 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:43.858 INFO:teuthology.orchestra.run.smithi012.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:22:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 8e:28:9a:60:ef:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:22:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::8c28:9aff:fe60:ef2c/64 scope link 2024-04-15T15:22:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:43.859 INFO:teuthology.orchestra.run.smithi012.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:22:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 36:f2:47:76:c7:11 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:22:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::34f2:47ff:fe76:c711/64 scope link 2024-04-15T15:22:43.859 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:43.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:43.859 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:22:43.923 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-04-15T15:22:43.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:43.925 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-04-15T15:22:43.981 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-04-15T15:22:43.982 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:22:43.982 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:22:43.982 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:22:43.982 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:22:43.982 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:22:43.982 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:22:43.982 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:22:43.982 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:22:43.983 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:22:43.983 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:22:44.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:22:44.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:22:44.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:44.121 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-15T15:22:44.138 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:22:44.138 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:22:44.138 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:22:44.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:44.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-04-15T15:22:44.207 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:22:44.207 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:22:44.207 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:22:44.263 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:22:44.263 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:22:44.263 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:22:44.318 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:22:44.387 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:22:44.444 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:22:44.445 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:22:44.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:44.447 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:22:44.499 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:22:44.500 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-15T15:22:44.543 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:22:44.571 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:22:44.627 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:22:44.627 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:22:44.627 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:22:44.627 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:22:44.627 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:22:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:22:44.629 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:22:44.630 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:22:44.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:44.630 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:22:44.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:2024-04-15T15:22:44.629+0000 7f4ea4f7c480 -1 init, newargv = 0x55d4595f6f40 newargc=15 2024-04-15T15:22:44.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[54041]: starting ceph client 2024-04-15T15:22:44.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[54041]: starting fuse 2024-04-15T15:22:44.647 INFO:teuthology.orchestra.run.smithi012.stdout:43 2024-04-15T15:22:44.647 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:22:44.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:22:44.647 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-04-15T15:22:44.647 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-04-15T15:22:44.647 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-04-15T15:22:44.648 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.0")) 2024-04-15T15:22:44.649 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-04-15T15:22:44.770 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.0.54041.asok 2024-04-15T15:22:44.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:22:44.781 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54041.asok 2024-04-15T15:22:44.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:44.781 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54041.asok status 2024-04-15T15:22:44.921 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:22:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-04-15T15:22:44.921 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "0", 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "54041", 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5709, 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-04-15T15:22:44.922 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5709 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3410145243 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3410145243 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:22:44.923 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.5709 192.168.0.1:0/3410145243", 2024-04-15T15:22:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.1:0/3410145243", 2024-04-15T15:22:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-04-15T15:22:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 18, 2024-04-15T15:22:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 39, 2024-04-15T15:22:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:22:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-04-15T15:22:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-04-15T15:22:44.924 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:22:44.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:44.930 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-04-15T15:22:44.930 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-15T15:22:44.945 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-04-15T15:22:44.945 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-15T15:22:44.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:44.946 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:22:45.015 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:22:45.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.157+0000 7fe7383c8640 1 Processor -- start 2024-04-15T15:22:45.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.157+0000 7fe7383c8640 1 -- start start 2024-04-15T15:22:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe7383c8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7300b3260 0x7fe7300b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe7383c8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe7300aefd0 0x7fe7300af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe7383c8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe73015a4f0 con 0x7fe73015a110 2024-04-15T15:22:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe7383c8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe7300ad500 con 0x7fe7300b3260 2024-04-15T15:22:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe7383c8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe7300b4d00 con 0x7fe7300aefd0 2024-04-15T15:22:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe735963640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7300b3260 0x7fe7300b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe735963640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe7300b3260 0x7fe7300b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49522/0 (socket says 172.21.15.12:49522) 2024-04-15T15:22:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe736164640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe73015a110 0x7fe73015d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42728/0 (socket says 172.21.15.12:42728) 2024-04-15T15:22:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe735963640 1 -- 172.21.15.12:0/1571440943 learned_addr learned my addr 172.21.15.12:0/1571440943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe736965640 1 --2- 172.21.15.12:0/1571440943 >> v2:172.21.15.55:3300/0 conn(0x7fe7300aefd0 0x7fe7300af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe735963640 1 -- 172.21.15.12:0/1571440943 >> v2:172.21.15.55:3300/0 conn(0x7fe7300aefd0 msgr2=0x7fe7300af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe735963640 1 --2- 172.21.15.12:0/1571440943 >> v2:172.21.15.55:3300/0 conn(0x7fe7300aefd0 0x7fe7300af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe735963640 1 -- 172.21.15.12:0/1571440943 >> v1:172.21.15.12:6789/0 conn(0x7fe73015a110 legacy=0x7fe73015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:45.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.159+0000 7fe735963640 1 -- 172.21.15.12:0/1571440943 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7300b3720 con 0x7fe7300b3260 2024-04-15T15:22:45.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.158+0000 7fe735162640 1 -- 172.21.15.12:0/1571440943 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 465984660 0 0) 0x7fe73015a4f0 con 0x7fe73015a110 2024-04-15T15:22:45.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.159+0000 7fe736965640 1 --2- 172.21.15.12:0/1571440943 >> v2:172.21.15.55:3300/0 conn(0x7fe7300aefd0 0x7fe7300af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:45.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.159+0000 7fe735963640 1 --2- 172.21.15.12:0/1571440943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300b3260 0x7fe7300b47c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe73015dc50 tx=0x7fe72402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18531f0610e76456 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:45.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.159+0000 7fe735162640 1 -- 172.21.15.12:0/1571440943 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe724034070 con 0x7fe7300b3260 2024-04-15T15:22:45.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.159+0000 7fe735162640 1 -- 172.21.15.12:0/1571440943 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe724030860 con 0x7fe7300b3260 2024-04-15T15:22:45.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.159+0000 7fe735162640 1 -- 172.21.15.12:0/1571440943 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe724030c30 con 0x7fe7300b3260 2024-04-15T15:22:45.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.160+0000 7fe7383c8640 1 -- 172.21.15.12:0/1571440943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300b3260 msgr2=0x7fe7300b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:45.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.160+0000 7fe7383c8640 1 --2- 172.21.15.12:0/1571440943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300b3260 0x7fe7300b47c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe73015dc50 tx=0x7fe72402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.160+0000 7fe7383c8640 1 -- 172.21.15.12:0/1571440943 shutdown_connections 2024-04-15T15:22:45.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.160+0000 7fe7383c8640 1 --2- 172.21.15.12:0/1571440943 >> v2:172.21.15.55:3300/0 conn(0x7fe7300aefd0 0x7fe7300af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.160+0000 7fe7383c8640 1 --2- 172.21.15.12:0/1571440943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300b3260 0x7fe7300b47c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.160+0000 7fe7383c8640 1 -- 172.21.15.12:0/1571440943 >> 172.21.15.12:0/1571440943 conn(0x7fe7300b1f60 msgr2=0x7fe7300b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:45.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.160+0000 7fe7383c8640 1 -- 172.21.15.12:0/1571440943 shutdown_connections 2024-04-15T15:22:45.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.160+0000 7fe7383c8640 1 -- 172.21.15.12:0/1571440943 wait complete. 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.161+0000 7fe7383c8640 1 Processor -- start 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.161+0000 7fe7383c8640 1 -- start start 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe7383c8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300aefd0 0x7fe73014fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe7383c8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe7300b3260 0x7fe7301f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe7383c8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe73015a110 0x7fe73015eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe7383c8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe7300b3fc0 con 0x7fe7300aefd0 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe7383c8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe7300b3ce0 con 0x7fe73015a110 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe7383c8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe7300af940 con 0x7fe7300b3260 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736164640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300aefd0 0x7fe73014fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736164640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300aefd0 0x7fe73014fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49526/0 (socket says 172.21.15.12:49526) 2024-04-15T15:22:45.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736164640 1 -- 172.21.15.12:0/3295896068 learned_addr learned my addr 172.21.15.12:0/3295896068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe735963640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe7300b3260 0x7fe7301f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736965640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe73015a110 0x7fe73015eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736164640 1 -- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe7300b3260 msgr2=0x7fe7301f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736164640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe7300b3260 0x7fe7301f3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736164640 1 -- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe73015a110 msgr2=0x7fe73015eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736164640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe73015a110 0x7fe73015eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.162+0000 7fe736164640 1 -- 172.21.15.12:0/3295896068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7301f43f0 con 0x7fe7300aefd0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.163+0000 7fe736164640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300aefd0 0x7fe73014fde0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c430 tx=0x7fe72800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.163+0000 7fe72effd640 1 -- 172.21.15.12:0/3295896068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7280074e0 con 0x7fe7300aefd0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.163+0000 7fe72effd640 1 -- 172.21.15.12:0/3295896068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7280047e0 con 0x7fe7300aefd0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.163+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe730073890 con 0x7fe7300aefd0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.163+0000 7fe72effd640 1 -- 172.21.15.12:0/3295896068 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe728004bd0 con 0x7fe7300aefd0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.163+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe730152410 con 0x7fe7300aefd0 2024-04-15T15:22:45.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.164+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7300689e0 con 0x7fe7300aefd0 2024-04-15T15:22:45.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.164+0000 7fe72effd640 1 -- 172.21.15.12:0/3295896068 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe728011070 con 0x7fe7300aefd0 2024-04-15T15:22:45.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.165+0000 7fe72effd640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe71c06c920 0x7fe71c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:45.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.165+0000 7fe72effd640 1 -- 172.21.15.12:0/3295896068 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe72808b850 con 0x7fe7300aefd0 2024-04-15T15:22:45.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.165+0000 7fe735963640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe71c06c920 0x7fe71c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:45.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.166+0000 7fe735963640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe71c06c920 0x7fe71c06ede0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe724036000 tx=0x7fe72403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:45.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.170+0000 7fe72effd640 1 -- 172.21.15.12:0/3295896068 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe72800e280 con 0x7fe7300aefd0 2024-04-15T15:22:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.397+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe730159510 con 0x7fe7300aefd0 2024-04-15T15:22:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.398+0000 7fe72effd640 1 -- 172.21.15.12:0/3295896068 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7fe728059b80 con 0x7fe7300aefd0 2024-04-15T15:22:45.400 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:22:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe71c06c920 msgr2=0x7fe71c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe71c06c920 0x7fe71c06ede0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe724036000 tx=0x7fe72403b040 comp rx=0 tx=0).stop 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300aefd0 msgr2=0x7fe73014fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300aefd0 0x7fe73014fde0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c430 tx=0x7fe72800c8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 shutdown_connections 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe71c06c920 0x7fe71c06ede0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe73015a110 0x7fe73015eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe7300b3260 0x7fe7301f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 --2- 172.21.15.12:0/3295896068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7300aefd0 0x7fe73014fde0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 >> 172.21.15.12:0/3295896068 conn(0x7fe7300b1f60 msgr2=0x7fe7300b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 shutdown_connections 2024-04-15T15:22:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:45.400+0000 7fe7383c8640 1 -- 172.21.15.12:0/3295896068 wait complete. 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi055.front.sepia.ceph.com 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:22:45.413 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:22:45.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:45.414 DEBUG:teuthology.orchestra.run.smithi055:> ip addr 2024-04-15T15:22:45.433 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:22:45.433 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:22:45.433 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:22:45.433 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:45.433 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f0 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f1 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:06 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f0 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.21.15.55/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft 42623sec preferred_lft 42623sec 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2024-04-15T15:22:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:07 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f1 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 2a:9f:ac:c6:f4:d0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::9cf3:7dff:fe9f:3592/64 scope link 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 2a:9f:ac:c6:f4:d0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:22:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::289f:acff:fec6:f4d0/64 scope link 2024-04-15T15:22:45.436 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:45.436 INFO:teuthology.orchestra.run.smithi055.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:22:45.436 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether fa:72:ab:ae:d3:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:22:45.436 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::f872:abff:feae:d3bd/64 scope link 2024-04-15T15:22:45.436 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:22:45.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:45.436 DEBUG:teuthology.orchestra.run.smithi055:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:22:45.505 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-04-15T15:22:45.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:45.507 DEBUG:teuthology.orchestra.run.smithi055:> route 2024-04-15T15:22:45.567 INFO:teuthology.orchestra.run.smithi055.stdout:Kernel IP routing table 2024-04-15T15:22:45.567 INFO:teuthology.orchestra.run.smithi055.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:22:45.567 INFO:teuthology.orchestra.run.smithi055.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:22:45.567 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:22:45.568 INFO:teuthology.orchestra.run.smithi055.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:22:45.568 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:22:45.568 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:22:45.568 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:22:45.568 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:22:45.568 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:22:45.568 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:22:45.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:22:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:22:45.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:22:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:22:45.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:45.717 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-04-15T15:22:45.733 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:22:45.733 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:22:45.734 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:22:45.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:45.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set brx.0 up 2024-04-15T15:22:45.803 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:22:45.803 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:22:45.803 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:22:45.859 INFO:teuthology.orchestra.run.smithi055.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:22:45.859 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:22:45.859 DEBUG:teuthology.orchestra.run.smithi055:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:22:45.914 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:22:45.983 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:22:46.040 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:22:46.041 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:22:46.042 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:22:46.043 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:22:46.043 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:22:46.043 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:22:46.043 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:22:46.043 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:22:46.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:46.043 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:22:46.096 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:22:46.096 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-15T15:22:46.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:22:46.167 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:22:46.225 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:22:46.225 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:22:46.225 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:22:46.225 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:22:46.226 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:22:46.226 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:22:46.226 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:22:46.226 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:22:46.226 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:22:46.226 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:22:46.226 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:22:46.226 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:22:46.227 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:22:46.227 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:22:46.227 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:22:46.227 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:22:46.227 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:22:46.227 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:22:46.227 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:22:46.228 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:22:46.228 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:22:46.228 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:22:46.228 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:22:46.228 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:22:46.228 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:22:46.228 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:22:46.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:46.229 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:22:46.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:2024-04-15T15:22:46.227+0000 7fd8bd071480 -1 init, newargv = 0x5647da301ff0 newargc=15 2024-04-15T15:22:46.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:ceph-fuse[45624]: starting ceph client 2024-04-15T15:22:46.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:ceph-fuse[45624]: starting fuse 2024-04-15T15:22:46.246 INFO:teuthology.orchestra.run.smithi055.stdout:41 2024-04-15T15:22:46.246 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-15T15:22:46.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:22:46.246 DEBUG:teuthology.orchestra.run.smithi055:> import glob 2024-04-15T15:22:46.246 DEBUG:teuthology.orchestra.run.smithi055:> import re 2024-04-15T15:22:46.246 DEBUG:teuthology.orchestra.run.smithi055:> import os 2024-04-15T15:22:46.246 DEBUG:teuthology.orchestra.run.smithi055:> import subprocess 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> def _find_admin_socket(client_name): 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> files = glob.glob(asok_path) 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> # Given a non-glob path, it better be there 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> if "*" not in asok_path: 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> assert(len(files) == 1) 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> return files[0] 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> for f in files: 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:22:46.247 DEBUG:teuthology.orchestra.run.smithi055:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:22:46.248 DEBUG:teuthology.orchestra.run.smithi055:> contents = proc_f.read() 2024-04-15T15:22:46.248 DEBUG:teuthology.orchestra.run.smithi055:> if mountpoint in contents: 2024-04-15T15:22:46.248 DEBUG:teuthology.orchestra.run.smithi055:> return f 2024-04-15T15:22:46.248 DEBUG:teuthology.orchestra.run.smithi055:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:22:46.248 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:22:46.248 DEBUG:teuthology.orchestra.run.smithi055:> print(_find_admin_socket("client.1")) 2024-04-15T15:22:46.248 DEBUG:teuthology.orchestra.run.smithi055:> ' 2024-04-15T15:22:46.368 INFO:teuthology.orchestra.run.smithi055.stdout:/var/run/ceph/ceph-client.1.45624.asok 2024-04-15T15:22:46.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:22:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:22:46.379 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45624.asok 2024-04-15T15:22:46.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:46.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45624.asok status 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "metadata": { 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "entity_id": "1", 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "hostname": "smithi055", 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "pid": "45624", 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "root": "/" 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_count": 0, 2024-04-15T15:22:46.519 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_pinned_count": 0, 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "id": 5353, 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "inst": { 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "name": { 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "client", 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "num": 5353 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3678955048 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:22:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3678955048 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "inst_str": "client.5353 192.168.0.1:0/3678955048", 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "addr_str": "192.168.0.1:0/3678955048", 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "inode_count": 1, 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_epoch": 18, 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch": 39, 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklisted": false, 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout: "fs_name": "cephfs" 2024-04-15T15:22:46.521 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-15T15:22:46.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:46.527 DEBUG:teuthology.orchestra.run.smithi055:> stat --file-system '--printf=%T 2024-04-15T15:22:46.527 DEBUG:teuthology.orchestra.run.smithi055:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-15T15:22:46.543 INFO:teuthology.orchestra.run.smithi055.stdout:fuseblk 2024-04-15T15:22:46.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-15T15:22:46.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:46.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:22:46.612 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 25 2024-04-15T15:22:46.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.729+0000 7f66f9d74640 1 Processor -- start 2024-04-15T15:22:46.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.729+0000 7f66f9d74640 1 -- start start 2024-04-15T15:22:46.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.730+0000 7f66f9d74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f40d2360 0x7f66f40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:46.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.730+0000 7f66f9d74640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66f40b47a0 con 0x7f66f415eae0 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.730+0000 7f66f9d74640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66f40b4a80 con 0x7f66f415e700 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.730+0000 7f66f9d74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f66f40b53b0 con 0x7f66f40d2360 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.730+0000 7f66f3fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f66f415eae0 0x7f66f40cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42742/0 (socket says 172.21.15.12:42742) 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.730+0000 7f66f3fff640 1 -- 172.21.15.12:0/551758078 learned_addr learned my addr 172.21.15.12:0/551758078 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f2ffd640 1 --2- 172.21.15.12:0/551758078 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f40d2360 0x7f66f40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f27fc640 1 -- 172.21.15.12:0/551758078 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270468554 0 0) 0x7f66f40b47a0 con 0x7f66f415eae0 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f27fc640 1 -- 172.21.15.12:0/551758078 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66dc003520 con 0x7f66f415eae0 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f27fc640 1 -- 172.21.15.12:0/551758078 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1556948722 0 0) 0x7f66f40b4a80 con 0x7f66f415e700 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f27fc640 1 -- 172.21.15.12:0/551758078 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66f40b47a0 con 0x7f66f415e700 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f2ffd640 1 -- 172.21.15.12:0/551758078 >> v1:172.21.15.55:6789/0 conn(0x7f66f415e700 legacy=0x7f66f40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f27fc640 1 -- 172.21.15.12:0/551758078 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2986798594 0 0) 0x7f66dc003520 con 0x7f66f415eae0 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f2ffd640 1 -- 172.21.15.12:0/551758078 >> v1:172.21.15.12:6789/0 conn(0x7f66f415eae0 legacy=0x7f66f40cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:46.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.731+0000 7f66f2ffd640 1 -- 172.21.15.12:0/551758078 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f40b4d60 con 0x7f66f40d2360 2024-04-15T15:22:46.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.732+0000 7f66f2ffd640 1 --2- 172.21.15.12:0/551758078 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f40d2360 0x7f66f40d2b60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f66f40b5530 tx=0x7f66e802b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1092e365f4d34023 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:46.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.732+0000 7f66f27fc640 1 -- 172.21.15.12:0/551758078 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8039070 con 0x7f66f40d2360 2024-04-15T15:22:46.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.732+0000 7f66f27fc640 1 -- 172.21.15.12:0/551758078 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e8030070 con 0x7f66f40d2360 2024-04-15T15:22:46.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.732+0000 7f66f27fc640 1 -- 172.21.15.12:0/551758078 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec001670 con 0x7f66f40d2360 2024-04-15T15:22:46.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.733+0000 7f66f9d74640 1 -- 172.21.15.12:0/551758078 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f40d2360 msgr2=0x7f66f40d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:46.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.733+0000 7f66f9d74640 1 --2- 172.21.15.12:0/551758078 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f40d2360 0x7f66f40d2b60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f66f40b5530 tx=0x7f66e802b7f0 comp rx=0 tx=0).stop 2024-04-15T15:22:46.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.733+0000 7f66f9d74640 1 -- 172.21.15.12:0/551758078 shutdown_connections 2024-04-15T15:22:46.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.733+0000 7f66f9d74640 1 --2- 172.21.15.12:0/551758078 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f40d2360 0x7f66f40d2b60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:46.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.733+0000 7f66f9d74640 1 -- 172.21.15.12:0/551758078 >> 172.21.15.12:0/551758078 conn(0x7f66f408bf60 msgr2=0x7f66f40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:46.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.733+0000 7f66f9d74640 1 -- 172.21.15.12:0/551758078 shutdown_connections 2024-04-15T15:22:46.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.733+0000 7f66f9d74640 1 -- 172.21.15.12:0/551758078 wait complete. 2024-04-15T15:22:46.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.734+0000 7f66f9d74640 1 Processor -- start 2024-04-15T15:22:46.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.734+0000 7f66f9d74640 1 -- start start 2024-04-15T15:22:46.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f9d74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66f40d2360 0x7f66f4160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:46.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f9d74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f415e700 0x7f66f41f5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:46.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f9d74640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f415eae0 0x7f66f41f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:46.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f9d74640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f66f40d1d40 con 0x7f66f40d2360 2024-04-15T15:22:46.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f9d74640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f66f40b4450 con 0x7f66f415eae0 2024-04-15T15:22:46.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f9d74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f66f40d1ec0 con 0x7f66f415e700 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66f40d2360 0x7f66f4160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66f40d2360 0x7f66f4160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49538/0 (socket says 172.21.15.12:49538) 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 -- 172.21.15.12:0/1122866950 learned_addr learned my addr 172.21.15.12:0/1122866950 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f3fff640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f415eae0 0x7f66f41f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f2ffd640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f415e700 0x7f66f41f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 -- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f415e700 msgr2=0x7f66f41f5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f415e700 0x7f66f41f5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 -- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f415eae0 msgr2=0x7f66f41f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f415eae0 0x7f66f41f4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 -- 172.21.15.12:0/1122866950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f41f5960 con 0x7f66f40d2360 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f3fff640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f415eae0 0x7f66f41f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f2ffd640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f415e700 0x7f66f41f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:46.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.735+0000 7f66f37fe640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66f40d2360 0x7f66f4160020 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f66e400c870 tx=0x7f66e400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.736+0000 7f66dbfff640 1 -- 172.21.15.12:0/1122866950 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4003bc0 con 0x7f66f40d2360 2024-04-15T15:22:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.736+0000 7f66dbfff640 1 -- 172.21.15.12:0/1122866950 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e4003d60 con 0x7f66f40d2360 2024-04-15T15:22:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.736+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f40cc8d0 con 0x7f66f40d2360 2024-04-15T15:22:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.736+0000 7f66dbfff640 1 -- 172.21.15.12:0/1122866950 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e400aa30 con 0x7f66f40d2360 2024-04-15T15:22:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.736+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f4160aa0 con 0x7f66f40d2360 2024-04-15T15:22:46.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.737+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f40689e0 con 0x7f66f40d2360 2024-04-15T15:22:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.737+0000 7f66dbfff640 1 -- 172.21.15.12:0/1122866950 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f66e400abd0 con 0x7f66f40d2360 2024-04-15T15:22:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.738+0000 7f66dbfff640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f66e006c920 0x7f66e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.738+0000 7f66dbfff640 1 -- 172.21.15.12:0/1122866950 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f66e408bc50 con 0x7f66f40d2360 2024-04-15T15:22:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.738+0000 7f66f2ffd640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f66e006c920 0x7f66e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:46.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.739+0000 7f66f2ffd640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f66e006c920 0x7f66e006ede0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f66e8036000 tx=0x7f66e8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:46.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.742+0000 7f66dbfff640 1 -- 172.21.15.12:0/1122866950 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66e4091020 con 0x7f66f40d2360 2024-04-15T15:22:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.928+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f66f40ba500 con 0x7f66f40d2360 2024-04-15T15:22:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.931+0000 7f66dbfff640 1 -- 172.21.15.12:0/1122866950 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v11)=0 v11) ==== 108+0+0 (secure 0 0 0) 0x7f66e4059f00 con 0x7f66f40d2360 2024-04-15T15:22:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f66e006c920 msgr2=0x7f66e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f66e006c920 0x7f66e006ede0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f66e8036000 tx=0x7f66e8008030 comp rx=0 tx=0).stop 2024-04-15T15:22:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66f40d2360 msgr2=0x7f66f4160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66f40d2360 0x7f66f4160020 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f66e400c870 tx=0x7f66e400cd30 comp rx=0 tx=0).stop 2024-04-15T15:22:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 shutdown_connections 2024-04-15T15:22:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f66e006c920 0x7f66e006ede0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f415eae0 0x7f66f41f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f66f415e700 0x7f66f41f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 --2- 172.21.15.12:0/1122866950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66f40d2360 0x7f66f4160020 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 >> 172.21.15.12:0/1122866950 conn(0x7f66f408bf60 msgr2=0x7f66f40cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 shutdown_connections 2024-04-15T15:22:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:46.933+0000 7f66f9d74640 1 -- 172.21.15.12:0/1122866950 wait complete. 2024-04-15T15:22:46.946 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_cache_quiesce_splitauth true 2024-04-15T15:22:47.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.146+0000 7f7121f3d640 1 Processor -- start 2024-04-15T15:22:47.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.147+0000 7f7121f3d640 1 -- start start 2024-04-15T15:22:47.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.147+0000 7f7121f3d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f711c09b880 0x7f711c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.147+0000 7f7121f3d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711c0d05e0 con 0x7f711c0d0da0 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.147+0000 7f7121f3d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711c071260 con 0x7f711c070940 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.147+0000 7f7121f3d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f711c09bce0 con 0x7f711c09b880 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.147+0000 7f711bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f711c0d0da0 0x7f711c0d11d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42754/0 (socket says 172.21.15.12:42754) 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.147+0000 7f711bfff640 1 -- 172.21.15.12:0/1793604601 learned_addr learned my addr 172.21.15.12:0/1793604601 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.148+0000 7f711affd640 1 --2- 172.21.15.12:0/1793604601 >> v2:172.21.15.55:3300/0 conn(0x7f711c09b880 0x7f711c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.148+0000 7f711a7fc640 1 -- 172.21.15.12:0/1793604601 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728295654 0 0) 0x7f711c0d05e0 con 0x7f711c0d0da0 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.148+0000 7f711a7fc640 1 -- 172.21.15.12:0/1793604601 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7104003520 con 0x7f711c0d0da0 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.148+0000 7f711affd640 1 -- 172.21.15.12:0/1793604601 >> v1:172.21.15.55:6789/0 conn(0x7f711c070940 legacy=0x7f711c160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.148+0000 7f711a7fc640 1 -- 172.21.15.12:0/1793604601 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1341391999 0 0) 0x7f7104003520 con 0x7f711c0d0da0 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.148+0000 7f711affd640 1 -- 172.21.15.12:0/1793604601 >> v1:172.21.15.12:6789/0 conn(0x7f711c0d0da0 legacy=0x7f711c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.148+0000 7f711affd640 1 -- 172.21.15.12:0/1793604601 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711c15f720 con 0x7f711c09b880 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.149+0000 7f711affd640 1 --2- 172.21.15.12:0/1793604601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c09b880 0x7f711c070d20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f71100097a0 tx=0x7f711002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ccdf4ee17e4484 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.149+0000 7f711a7fc640 1 -- 172.21.15.12:0/1793604601 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110038070 con 0x7f711c09b880 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.149+0000 7f711a7fc640 1 -- 172.21.15.12:0/1793604601 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7110031500 con 0x7f711c09b880 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.149+0000 7f711a7fc640 1 -- 172.21.15.12:0/1793604601 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71140019f0 con 0x7f711c09b880 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.149+0000 7f7121f3d640 1 -- 172.21.15.12:0/1793604601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c09b880 msgr2=0x7f711c070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.149+0000 7f7121f3d640 1 --2- 172.21.15.12:0/1793604601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c09b880 0x7f711c070d20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f71100097a0 tx=0x7f711002b820 comp rx=0 tx=0).stop 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.150+0000 7f7121f3d640 1 -- 172.21.15.12:0/1793604601 shutdown_connections 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.150+0000 7f7121f3d640 1 --2- 172.21.15.12:0/1793604601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c09b880 0x7f711c070d20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.150+0000 7f7121f3d640 1 -- 172.21.15.12:0/1793604601 >> 172.21.15.12:0/1793604601 conn(0x7f711c0b3f40 msgr2=0x7f711c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.150+0000 7f7121f3d640 1 -- 172.21.15.12:0/1793604601 shutdown_connections 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.150+0000 7f7121f3d640 1 -- 172.21.15.12:0/1793604601 wait complete. 2024-04-15T15:22:47.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.151+0000 7f7121f3d640 1 Processor -- start 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.151+0000 7f7121f3d640 1 -- start start 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7121f3d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c070940 0x7f711c0c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7121f3d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711c09b880 0x7f711c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7121f3d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f711c0d0da0 0x7f711c1f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7121f3d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f711c0d0880 con 0x7f711c09b880 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7121f3d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f711c15fce0 con 0x7f711c070940 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7121f3d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f711c0d18e0 con 0x7f711c0d0da0 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711c09b880 0x7f711c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711c09b880 0x7f711c0c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49540/0 (socket says 172.21.15.12:49540) 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 -- 172.21.15.12:0/4118633928 learned_addr learned my addr 172.21.15.12:0/4118633928 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711b7fe640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c070940 0x7f711c0c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711bfff640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f711c0d0da0 0x7f711c1f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 -- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f711c0d0da0 msgr2=0x7f711c1f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f711c0d0da0 0x7f711c1f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 -- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c070940 msgr2=0x7f711c0c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c070940 0x7f711c0c8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 -- 172.21.15.12:0/4118633928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711c0b5660 con 0x7f711c09b880 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711b7fe640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c070940 0x7f711c0c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711bfff640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f711c0d0da0 0x7f711c1f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f711affd640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711c09b880 0x7f711c0c8980 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f711002b860 tx=0x7f71100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7103fff640 1 -- 172.21.15.12:0/4118633928 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110043070 con 0x7f711c09b880 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7103fff640 1 -- 172.21.15.12:0/4118633928 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711003f870 con 0x7f711c09b880 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7103fff640 1 -- 172.21.15.12:0/4118633928 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110038040 con 0x7f711c09b880 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.152+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711c0b5940 con 0x7f711c09b880 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.153+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711c0b5f00 con 0x7f711c09b880 2024-04-15T15:22:47.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.154+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711c0689e0 con 0x7f711c09b880 2024-04-15T15:22:47.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.154+0000 7f7103fff640 1 -- 172.21.15.12:0/4118633928 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7110046070 con 0x7f711c09b880 2024-04-15T15:22:47.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.154+0000 7f7103fff640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f710806c8f0 0x7f710806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.155+0000 7f7103fff640 1 -- 172.21.15.12:0/4118633928 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f71100ad910 con 0x7f711c09b880 2024-04-15T15:22:47.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.155+0000 7f711b7fe640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f710806c8f0 0x7f710806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.155+0000 7f711b7fe640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f710806c8f0 0x7f710806edb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f710c007fd0 tx=0x7f710c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.159+0000 7f7103fff640 1 -- 172.21.15.12:0/4118633928 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7110030070 con 0x7f711c09b880 2024-04-15T15:22:47.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.343+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mds_cache_quiesce_splitauth}] v 0) -- 0x7f711c070720 con 0x7f711c09b880 2024-04-15T15:22:47.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.346+0000 7f7103fff640 1 -- 172.21.15.12:0/4118633928 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_cache_quiesce_splitauth}]=0 v12)=0 v12) ==== 128+0+0 (secure 0 0 0) 0x7f711007bcd0 con 0x7f711c09b880 2024-04-15T15:22:47.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f710806c8f0 msgr2=0x7f710806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f710806c8f0 0x7f710806edb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f710c007fd0 tx=0x7f710c009040 comp rx=0 tx=0).stop 2024-04-15T15:22:47.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711c09b880 msgr2=0x7f711c0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711c09b880 0x7f711c0c8980 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f711002b860 tx=0x7f71100046c0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 shutdown_connections 2024-04-15T15:22:47.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f711c0d0da0 0x7f711c1f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f711c09b880 0x7f711c0c8980 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f710806c8f0 0x7f710806edb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 --2- 172.21.15.12:0/4118633928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f711c070940 0x7f711c0c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 >> 172.21.15.12:0/4118633928 conn(0x7f711c0b3f40 msgr2=0x7f711c15eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:47.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 shutdown_connections 2024-04-15T15:22:47.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.348+0000 7f7121f3d640 1 -- 172.21.15.12:0/4118633928 wait complete. 2024-04-15T15:22:47.362 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol_quiesce --mode=777 2024-04-15T15:22:47.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.467+0000 7faeafcf2640 1 Processor -- start 2024-04-15T15:22:47.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.467+0000 7faeafcf2640 1 -- start start 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeafcf2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faea8142f00 0x7faea809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeafcf2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea8145d70 0x7faea81461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeafcf2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7faea815cf40 0x7faea815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeafcf2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7faea8145040 con 0x7faea8142f00 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeafcf2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7faea81458e0 con 0x7faea815cf40 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeafcf2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faea81432e0 con 0x7faea8145d70 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeada8e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faea8142f00 0x7faea809a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeada8e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faea8142f00 0x7faea809a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49548/0 (socket says 172.21.15.12:49548) 2024-04-15T15:22:47.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.468+0000 7faeada8e640 1 -- 172.21.15.12:0/4066430089 learned_addr learned my addr 172.21.15.12:0/4066430089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:47.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.469+0000 7faeada8e640 1 -- 172.21.15.12:0/4066430089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea8145d70 msgr2=0x7faea81461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:47.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.469+0000 7faeada8e640 1 --2- 172.21.15.12:0/4066430089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea8145d70 0x7faea81461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.469+0000 7faeada8e640 1 -- 172.21.15.12:0/4066430089 >> v2:172.21.15.55:3300/0 conn(0x7faea815cf40 msgr2=0x7faea815d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:47.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.469+0000 7faeada8e640 1 --2- 172.21.15.12:0/4066430089 >> v2:172.21.15.55:3300/0 conn(0x7faea815cf40 0x7faea815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.469+0000 7faeada8e640 1 -- 172.21.15.12:0/4066430089 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea80b3d70 con 0x7faea8142f00 2024-04-15T15:22:47.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.469+0000 7faeada8e640 1 --2- 172.21.15.12:0/4066430089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8142f00 0x7faea809a9d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7faea8073dd0 tx=0x7fae9802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179e16e43969e56e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeaca8c640 1 -- 172.21.15.12:0/4066430089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9802ca70 con 0x7faea8142f00 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeaca8c640 1 -- 172.21.15.12:0/4066430089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae9802cc10 con 0x7faea8142f00 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeaca8c640 1 -- 172.21.15.12:0/4066430089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae980343f0 con 0x7faea8142f00 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 -- 172.21.15.12:0/4066430089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8142f00 msgr2=0x7faea809a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 --2- 172.21.15.12:0/4066430089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8142f00 0x7faea809a9d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7faea8073dd0 tx=0x7fae9802bed0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 -- 172.21.15.12:0/4066430089 shutdown_connections 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 --2- 172.21.15.12:0/4066430089 >> v2:172.21.15.55:3300/0 conn(0x7faea815cf40 0x7faea815d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 --2- 172.21.15.12:0/4066430089 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea8145d70 0x7faea81461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 --2- 172.21.15.12:0/4066430089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8142f00 0x7faea809a9d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 -- 172.21.15.12:0/4066430089 >> 172.21.15.12:0/4066430089 conn(0x7faea80b6a20 msgr2=0x7faea80b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 -- 172.21.15.12:0/4066430089 shutdown_connections 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.470+0000 7faeafcf2640 1 -- 172.21.15.12:0/4066430089 wait complete. 2024-04-15T15:22:47.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.471+0000 7faeafcf2640 1 Processor -- start 2024-04-15T15:22:47.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.471+0000 7faeafcf2640 1 -- start start 2024-04-15T15:22:47.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faeafcf2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faea8142f00 0x7faea81f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faeafcf2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8145d70 0x7faea81f5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faeafcf2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea815cf40 0x7faea81f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faeafcf2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faea80b39b0 con 0x7faea8145d70 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faeafcf2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faea8145570 con 0x7faea8142f00 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faeafcf2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7faea80b3b30 con 0x7faea815cf40 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faead28d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8145d70 0x7faea81f5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faead28d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8145d70 0x7faea81f5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49554/0 (socket says 172.21.15.12:49554) 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faead28d640 1 -- 172.21.15.12:0/1106295059 learned_addr learned my addr 172.21.15.12:0/1106295059 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faeada8e640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faea8142f00 0x7faea81f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.472+0000 7faeae28f640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea815cf40 0x7faea81f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faead28d640 1 -- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea815cf40 msgr2=0x7faea81f5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faead28d640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea815cf40 0x7faea81f5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faead28d640 1 -- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faea8142f00 msgr2=0x7faea81f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faead28d640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faea8142f00 0x7faea81f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faead28d640 1 -- 172.21.15.12:0/1106295059 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea80cf970 con 0x7faea8145d70 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faeada8e640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faea8142f00 0x7faea81f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faeae28f640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea815cf40 0x7faea81f5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faead28d640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8145d70 0x7faea81f5120 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00b4a0 tx=0x7fae9c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faea67fc640 1 -- 172.21.15.12:0/1106295059 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c014070 con 0x7faea8145d70 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faea67fc640 1 -- 172.21.15.12:0/1106295059 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae9c010610 con 0x7faea8145d70 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea813f6e0 con 0x7faea8145d70 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faea67fc640 1 -- 172.21.15.12:0/1106295059 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c010a00 con 0x7faea8145d70 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.473+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea80cfc10 con 0x7faea8145d70 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.474+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea80689e0 con 0x7faea8145d70 2024-04-15T15:22:47.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.474+0000 7faea67fc640 1 -- 172.21.15.12:0/1106295059 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae9c017070 con 0x7faea8145d70 2024-04-15T15:22:47.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.475+0000 7faea67fc640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae9406c8f0 0x7fae9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.475+0000 7faea67fc640 1 -- 172.21.15.12:0/1106295059 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fae9c08b6a0 con 0x7faea8145d70 2024-04-15T15:22:47.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.475+0000 7faeada8e640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae9406c8f0 0x7fae9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.476+0000 7faeada8e640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae9406c8f0 0x7fae9406edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fae98035000 tx=0x7fae98030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.479+0000 7faea67fc640 1 -- 172.21.15.12:0/1106295059 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae9c0599d0 con 0x7faea8145d70 2024-04-15T15:22:47.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.691+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7faea80690f0 con 0x7fae9406c8f0 2024-04-15T15:22:47.699 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:22:47.698+0000 7fc675c4d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:22:47.699 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:22:47.698+0000 7fc675c4d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:22:47.699 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:22:47.698+0000 7fc675c4d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:22:47.699 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:22:47.698+0000 7fc675c4d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:22:47.699 INFO:tasks.ceph.mgr.x.smithi055.stderr:2024-04-15T15:22:47.698+0000 7fc675c4d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:22:47.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.732+0000 7faea67fc640 1 -- 172.21.15.12:0/1106295059 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faea80690f0 con 0x7fae9406c8f0 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae9406c8f0 msgr2=0x7fae9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae9406c8f0 0x7fae9406edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fae98035000 tx=0x7fae98030000 comp rx=0 tx=0).stop 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8145d70 msgr2=0x7faea81f5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8145d70 0x7faea81f5120 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00b4a0 tx=0x7fae9c00b960 comp rx=0 tx=0).stop 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 shutdown_connections 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7faea815cf40 0x7faea81f5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faea8145d70 0x7faea81f5120 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fae9406c8f0 0x7fae9406edb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 --2- 172.21.15.12:0/1106295059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faea8142f00 0x7faea81f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 >> 172.21.15.12:0/1106295059 conn(0x7faea80b6a20 msgr2=0x7faea815df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:47.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 shutdown_connections 2024-04-15T15:22:47.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.734+0000 7faeafcf2640 1 -- 172.21.15.12:0/1106295059 wait complete. 2024-04-15T15:22:47.747 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_quiesce 2024-04-15T15:22:47.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.853+0000 7f929e5a8640 1 Processor -- start 2024-04-15T15:22:47.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.853+0000 7f929e5a8640 1 -- start start 2024-04-15T15:22:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929e5a8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f929815ecb0 0x7f92980b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929e5a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f92980af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929e5a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92980b10d0 con 0x7f92980b4350 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929e5a8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f929815f090 con 0x7f929815ecb0 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929e5a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f92980cdf80 con 0x7f92980b1d30 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929cb45640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f92980af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f9297fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f929815ecb0 0x7f92980b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f92977fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f92980b4350 0x7f92980b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:36390/0 (socket says 172.21.15.12:36390) 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929cb45640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f92980af340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:55578/0 (socket says 172.21.15.12:55578) 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f92977fe640 1 -- 172.21.15.12:0/2192513552 learned_addr learned my addr 172.21.15.12:0/2192513552 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929cb45640 1 -- 172.21.15.12:0/2192513552 >> v2:172.21.15.55:3300/0 conn(0x7f929815ecb0 msgr2=0x7f92980b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929cb45640 1 --2- 172.21.15.12:0/2192513552 >> v2:172.21.15.55:3300/0 conn(0x7f929815ecb0 0x7f92980b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.854+0000 7f929cb45640 1 -- 172.21.15.12:0/2192513552 >> v1:172.21.15.55:6789/0 conn(0x7f92980b4350 legacy=0x7f92980b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.855+0000 7f929cb45640 1 -- 172.21.15.12:0/2192513552 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92980af920 con 0x7f92980b1d30 2024-04-15T15:22:47.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.855+0000 7f929cb45640 1 --2- 172.21.15.12:0/2192513552 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f92980af340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9290009d90 tx=0x7f929002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e457924e5131f0e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f9296ffd640 1 -- 172.21.15.12:0/2192513552 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290039070 con 0x7f92980b1d30 2024-04-15T15:22:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f9296ffd640 1 -- 172.21.15.12:0/2192513552 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f929002cce0 con 0x7f92980b1d30 2024-04-15T15:22:47.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f9296ffd640 1 -- 172.21.15.12:0/2192513552 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290034660 con 0x7f92980b1d30 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f929e5a8640 1 -- 172.21.15.12:0/2192513552 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 msgr2=0x7f92980af340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2192513552 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f92980af340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9290009d90 tx=0x7f929002bf30 comp rx=0 tx=0).stop 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f929e5a8640 1 -- 172.21.15.12:0/2192513552 shutdown_connections 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2192513552 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f92980af340 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2192513552 >> v2:172.21.15.55:3300/0 conn(0x7f929815ecb0 0x7f92980b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f929e5a8640 1 -- 172.21.15.12:0/2192513552 >> 172.21.15.12:0/2192513552 conn(0x7f92980b6870 msgr2=0x7f92980b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f929e5a8640 1 -- 172.21.15.12:0/2192513552 shutdown_connections 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.856+0000 7f929e5a8640 1 -- 172.21.15.12:0/2192513552 wait complete. 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.857+0000 7f929e5a8640 1 Processor -- start 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.857+0000 7f929e5a8640 1 -- start start 2024-04-15T15:22:47.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929e5a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f929815f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929e5a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92980b4350 0x7f92981f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929e5a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f929815ecb0 0x7f92980ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929e5a8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f92980b47c0 con 0x7f929815ecb0 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929e5a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f92980afee0 con 0x7f92980b4350 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929e5a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f92980d01c0 con 0x7f92980b1d30 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929cb45640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f929815ecb0 0x7f92980ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929cb45640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f929815ecb0 0x7f92980ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49570/0 (socket says 172.21.15.12:49570) 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f929cb45640 1 -- 172.21.15.12:0/2882577625 learned_addr learned my addr 172.21.15.12:0/2882577625 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f9297fff640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f929815f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.858+0000 7f92977fe640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92980b4350 0x7f92981f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f929cb45640 1 -- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 msgr2=0x7f929815f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f929cb45640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f929815f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f929cb45640 1 -- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92980b4350 msgr2=0x7f92981f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:47.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f929cb45640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92980b4350 0x7f92981f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f929cb45640 1 -- 172.21.15.12:0/2882577625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92980cf8b0 con 0x7f929815ecb0 2024-04-15T15:22:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f929cb45640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f929815ecb0 0x7f92980ccca0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9290009eb0 tx=0x7f9290004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f9294ff9640 1 -- 172.21.15.12:0/2882577625 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290039050 con 0x7f929815ecb0 2024-04-15T15:22:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f9294ff9640 1 -- 172.21.15.12:0/2882577625 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9290045070 con 0x7f929815ecb0 2024-04-15T15:22:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92980cd6b0 con 0x7f929815ecb0 2024-04-15T15:22:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f9294ff9640 1 -- 172.21.15.12:0/2882577625 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290009450 con 0x7f929815ecb0 2024-04-15T15:22:47.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.859+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92980cfe50 con 0x7f929815ecb0 2024-04-15T15:22:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.860+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92980689e0 con 0x7f929815ecb0 2024-04-15T15:22:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.860+0000 7f9294ff9640 1 -- 172.21.15.12:0/2882577625 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f929003e830 con 0x7f929815ecb0 2024-04-15T15:22:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.861+0000 7f9294ff9640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928406c920 0x7f928406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.861+0000 7f9294ff9640 1 -- 172.21.15.12:0/2882577625 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f92900ae7e0 con 0x7f929815ecb0 2024-04-15T15:22:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.861+0000 7f9297fff640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928406c920 0x7f928406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:47.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.861+0000 7f9297fff640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928406c920 0x7f928406ede0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9288009aa0 tx=0x7f9288009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:47.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:47.865+0000 7f9294ff9640 1 -- 172.21.15.12:0/2882577625 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f929007c590 con 0x7f929815ecb0 2024-04-15T15:22:48.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.085+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "target": ["mon-mgr", ""]}) -- 0x7f929806c7c0 con 0x7f928406c920 2024-04-15T15:22:48.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.091+0000 7f9294ff9640 1 -- 172.21.15.12:0/2882577625 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+70 (secure 0 0 0) 0x7f929806c7c0 con 0x7f928406c920 2024-04-15T15:22:48.093 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a 2024-04-15T15:22:48.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928406c920 msgr2=0x7f928406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928406c920 0x7f928406ede0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9288009aa0 tx=0x7f9288009190 comp rx=0 tx=0).stop 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f929815ecb0 msgr2=0x7f92980ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f929815ecb0 0x7f92980ccca0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9290009eb0 tx=0x7f9290004a40 comp rx=0 tx=0).stop 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 shutdown_connections 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f929815ecb0 0x7f92980ccca0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f928406c920 0x7f928406ede0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92980b4350 0x7f92981f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 --2- 172.21.15.12:0/2882577625 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f92980b1d30 0x7f929815f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 >> 172.21.15.12:0/2882577625 conn(0x7f92980b6870 msgr2=0x7f92980af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 shutdown_connections 2024-04-15T15:22:48.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.092+0000 7f929e5a8640 1 -- 172.21.15.12:0/2882577625 wait complete. 2024-04-15T15:22:48.106 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:22:48.106 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:48.121 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.263+0000 7fe6f380d640 1 Processor -- start 2024-04-15T15:22:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.264+0000 7fe6f380d640 1 -- start start 2024-04-15T15:22:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.264+0000 7fe6f380d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe6ec15e700 0x7fe6ec0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.264+0000 7fe6f380d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec0d2360 0x7fe6ec0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.264+0000 7fe6f380d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ec0b47a0 con 0x7fe6ec15eae0 2024-04-15T15:22:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.264+0000 7fe6f380d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe6ec0b53b0 con 0x7fe6ec15e700 2024-04-15T15:22:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.264+0000 7fe6f380d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ec0b4a80 con 0x7fe6ec0d2360 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f15a9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe6ec15e700 0x7fe6ec0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f1daa640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fe6ec15eae0 0x7fe6ec0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:36392/0 (socket says 172.21.15.12:36392) 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f0da8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec0d2360 0x7fe6ec0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f1daa640 1 -- 172.21.15.12:0/2238136138 learned_addr learned my addr 172.21.15.12:0/2238136138 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f0da8640 1 -- 172.21.15.12:0/2238136138 >> v2:172.21.15.55:3300/0 conn(0x7fe6ec15e700 msgr2=0x7fe6ec0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f0da8640 1 --2- 172.21.15.12:0/2238136138 >> v2:172.21.15.55:3300/0 conn(0x7fe6ec15e700 0x7fe6ec0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f0da8640 1 -- 172.21.15.12:0/2238136138 >> v1:172.21.15.55:6789/0 conn(0x7fe6ec15eae0 legacy=0x7fe6ec0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f0da8640 1 -- 172.21.15.12:0/2238136138 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec0b4d60 con 0x7fe6ec0d2360 2024-04-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.265+0000 7fe6f15a9640 1 --2- 172.21.15.12:0/2238136138 >> v2:172.21.15.55:3300/0 conn(0x7fe6ec15e700 0x7fe6ec0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.266+0000 7fe6f0da8640 1 --2- 172.21.15.12:0/2238136138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec0d2360 0x7fe6ec0d2b60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0097a0 tx=0x7fe6dc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e62626cc8ce59ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.266+0000 7fe6e3fff640 1 -- 172.21.15.12:0/2238136138 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc035020 con 0x7fe6ec0d2360 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.266+0000 7fe6e3fff640 1 -- 172.21.15.12:0/2238136138 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6dc031830 con 0x7fe6ec0d2360 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.266+0000 7fe6e3fff640 1 -- 172.21.15.12:0/2238136138 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc031c00 con 0x7fe6ec0d2360 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.267+0000 7fe6f380d640 1 -- 172.21.15.12:0/2238136138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec0d2360 msgr2=0x7fe6ec0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.267+0000 7fe6f380d640 1 --2- 172.21.15.12:0/2238136138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec0d2360 0x7fe6ec0d2b60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0097a0 tx=0x7fe6dc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.267+0000 7fe6f380d640 1 -- 172.21.15.12:0/2238136138 shutdown_connections 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.267+0000 7fe6f380d640 1 --2- 172.21.15.12:0/2238136138 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec0d2360 0x7fe6ec0d2b60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.267+0000 7fe6f380d640 1 --2- 172.21.15.12:0/2238136138 >> v2:172.21.15.55:3300/0 conn(0x7fe6ec15e700 0x7fe6ec0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.267+0000 7fe6f380d640 1 -- 172.21.15.12:0/2238136138 >> 172.21.15.12:0/2238136138 conn(0x7fe6ec08bf60 msgr2=0x7fe6ec0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.267+0000 7fe6f380d640 1 -- 172.21.15.12:0/2238136138 shutdown_connections 2024-04-15T15:22:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.267+0000 7fe6f380d640 1 -- 172.21.15.12:0/2238136138 wait complete. 2024-04-15T15:22:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.268+0000 7fe6f380d640 1 Processor -- start 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.268+0000 7fe6f380d640 1 -- start start 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f380d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6ec0d2360 0x7fe6ec0b84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f380d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6ec15e700 0x7fe6ec0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f380d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec15eae0 0x7fe6ec160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f380d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec0b5d80 con 0x7fe6ec0d2360 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f380d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec0b4450 con 0x7fe6ec15e700 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f380d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ec0d1d40 con 0x7fe6ec15eae0 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6ec0d2360 0x7fe6ec0b84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6ec0d2360 0x7fe6ec0b84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49582/0 (socket says 172.21.15.12:49582) 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 -- 172.21.15.12:0/769183133 learned_addr learned my addr 172.21.15.12:0/769183133 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f0da8640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6ec15e700 0x7fe6ec0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f1daa640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec15eae0 0x7fe6ec160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 -- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec15eae0 msgr2=0x7fe6ec160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec15eae0 0x7fe6ec160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 -- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6ec15e700 msgr2=0x7fe6ec0c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6ec15e700 0x7fe6ec0c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 -- 172.21.15.12:0/769183133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec0caed0 con 0x7fe6ec0d2360 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f1daa640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec15eae0 0x7fe6ec160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f15a9640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6ec0d2360 0x7fe6ec0b84a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400ec10 tx=0x7fe6e400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6e1ffb640 1 -- 172.21.15.12:0/769183133 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e400cc30 con 0x7fe6ec0d2360 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6e1ffb640 1 -- 172.21.15.12:0/769183133 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e400cdd0 con 0x7fe6ec0d2360 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ec0cabf0 con 0x7fe6ec0d2360 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.269+0000 7fe6e1ffb640 1 -- 172.21.15.12:0/769183133 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4005bb0 con 0x7fe6ec0d2360 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.270+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ec0b7f80 con 0x7fe6ec0d2360 2024-04-15T15:22:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.271+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6ec0689e0 con 0x7fe6ec0d2360 2024-04-15T15:22:48.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.271+0000 7fe6e1ffb640 1 -- 172.21.15.12:0/769183133 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe6e4006030 con 0x7fe6ec0d2360 2024-04-15T15:22:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.271+0000 7fe6e1ffb640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c8f0 0x7fe6d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.272+0000 7fe6e1ffb640 1 -- 172.21.15.12:0/769183133 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe6e408c1c0 con 0x7fe6ec0d2360 2024-04-15T15:22:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.272+0000 7fe6f0da8640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c8f0 0x7fe6d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.272+0000 7fe6f0da8640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c8f0 0x7fe6d006edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc036000 tx=0x7fe6dc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.276+0000 7fe6e1ffb640 1 -- 172.21.15.12:0/769183133 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6e4017020 con 0x7fe6ec0d2360 2024-04-15T15:22:48.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.465+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6ec0ba500 con 0x7fe6ec0d2360 2024-04-15T15:22:48.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.466+0000 7fe6e1ffb640 1 -- 172.21.15.12:0/769183133 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16906 (secure 0 0 0) 0x7fe6e405a460 con 0x7fe6ec0d2360 2024-04-15T15:22:48.468 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:48.469 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":39,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:22:34.287725+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:48.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c8f0 msgr2=0x7fe6d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c8f0 0x7fe6d006edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc036000 tx=0x7fe6dc008030 comp rx=0 tx=0).stop 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6ec0d2360 msgr2=0x7fe6ec0b84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6ec0d2360 0x7fe6ec0b84a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400ec10 tx=0x7fe6e400c520 comp rx=0 tx=0).stop 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 shutdown_connections 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6ec15eae0 0x7fe6ec160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe6d006c8f0 0x7fe6d006edb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6ec15e700 0x7fe6ec0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 --2- 172.21.15.12:0/769183133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6ec0d2360 0x7fe6ec0b84a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 >> 172.21.15.12:0/769183133 conn(0x7fe6ec08bf60 msgr2=0x7fe6ec0d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 shutdown_connections 2024-04-15T15:22:48.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.468+0000 7fe6f380d640 1 -- 172.21.15.12:0/769183133 wait complete. 2024-04-15T15:22:48.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:22:48.482 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:48.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.586+0000 7fc0e1950640 1 Processor -- start 2024-04-15T15:22:48.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.587+0000 7fc0e1950640 1 -- start start 2024-04-15T15:22:48.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.587+0000 7fc0e1950640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc0dc15e700 0x7fc0dc0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.587+0000 7fc0e1950640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d3270 0x7fc0dc0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.587+0000 7fc0e1950640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0dc0b47a0 con 0x7fc0dc0d2360 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.587+0000 7fc0e1950640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc0dc0b53b0 con 0x7fc0dc15e700 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.587+0000 7fc0e1950640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b4a80 con 0x7fc0dc0d3270 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0db7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d3270 0x7fc0dc0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0daffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc0dc15e700 0x7fc0dc0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0db7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d3270 0x7fc0dc0cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:55598/0 (socket says 172.21.15.12:55598) 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0da7fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc0dc0d2360 0x7fc0dc0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:36406/0 (socket says 172.21.15.12:36406) 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0db7fe640 1 -- 172.21.15.12:0/2380406092 learned_addr learned my addr 172.21.15.12:0/2380406092 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0db7fe640 1 -- 172.21.15.12:0/2380406092 >> v2:172.21.15.55:3300/0 conn(0x7fc0dc15e700 msgr2=0x7fc0dc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0db7fe640 1 --2- 172.21.15.12:0/2380406092 >> v2:172.21.15.55:3300/0 conn(0x7fc0dc15e700 0x7fc0dc0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0db7fe640 1 -- 172.21.15.12:0/2380406092 >> v1:172.21.15.55:6789/0 conn(0x7fc0dc0d2360 legacy=0x7fc0dc0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.588+0000 7fc0db7fe640 1 -- 172.21.15.12:0/2380406092 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc0b4d60 con 0x7fc0dc0d3270 2024-04-15T15:22:48.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.589+0000 7fc0db7fe640 1 --2- 172.21.15.12:0/2380406092 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d3270 0x7fc0dc0cf370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009e50 tx=0x7fc0cc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0597626a16809a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.589+0000 7fc0d9ffb640 1 -- 172.21.15.12:0/2380406092 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc02cb00 con 0x7fc0dc0d3270 2024-04-15T15:22:48.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.589+0000 7fc0d9ffb640 1 -- 172.21.15.12:0/2380406092 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0cc02cca0 con 0x7fc0dc0d3270 2024-04-15T15:22:48.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.589+0000 7fc0d9ffb640 1 -- 172.21.15.12:0/2380406092 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc034610 con 0x7fc0dc0d3270 2024-04-15T15:22:48.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.590+0000 7fc0e1950640 1 -- 172.21.15.12:0/2380406092 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d3270 msgr2=0x7fc0dc0cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.590+0000 7fc0e1950640 1 --2- 172.21.15.12:0/2380406092 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d3270 0x7fc0dc0cf370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009e50 tx=0x7fc0cc02bf60 comp rx=0 tx=0).stop 2024-04-15T15:22:48.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.590+0000 7fc0e1950640 1 -- 172.21.15.12:0/2380406092 shutdown_connections 2024-04-15T15:22:48.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.590+0000 7fc0e1950640 1 --2- 172.21.15.12:0/2380406092 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d3270 0x7fc0dc0cf370 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.590+0000 7fc0e1950640 1 --2- 172.21.15.12:0/2380406092 >> v2:172.21.15.55:3300/0 conn(0x7fc0dc15e700 0x7fc0dc0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.590+0000 7fc0e1950640 1 -- 172.21.15.12:0/2380406092 >> 172.21.15.12:0/2380406092 conn(0x7fc0dc08bf60 msgr2=0x7fc0dc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:48.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.590+0000 7fc0e1950640 1 -- 172.21.15.12:0/2380406092 shutdown_connections 2024-04-15T15:22:48.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.590+0000 7fc0e1950640 1 -- 172.21.15.12:0/2380406092 wait complete. 2024-04-15T15:22:48.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.591+0000 7fc0e1950640 1 Processor -- start 2024-04-15T15:22:48.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.591+0000 7fc0e1950640 1 -- start start 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0e1950640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d2360 0x7fc0dc1548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0e1950640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc0d3270 0x7fc0dc15ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0e1950640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0dc15e700 0x7fc0dc0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0e1950640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b5d80 con 0x7fc0dc0d3270 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0e1950640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b4450 con 0x7fc0dc15e700 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0e1950640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc0dc0d1d40 con 0x7fc0dc0d2360 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc0d3270 0x7fc0dc15ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc0d3270 0x7fc0dc15ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49590/0 (socket says 172.21.15.12:49590) 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 -- 172.21.15.12:0/3793744771 learned_addr learned my addr 172.21.15.12:0/3793744771 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0db7fe640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0dc15e700 0x7fc0dc0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0daffd640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d2360 0x7fc0dc1548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 -- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d2360 msgr2=0x7fc0dc1548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d2360 0x7fc0dc1548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 -- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0dc15e700 msgr2=0x7fc0dc0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0dc15e700 0x7fc0dc0c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 -- 172.21.15.12:0/3793744771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc1f4740 con 0x7fc0dc0d3270 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0da7fc640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc0d3270 0x7fc0dc15ec30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400c3c0 tx=0x7fc0c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.592+0000 7fc0d3fff640 1 -- 172.21.15.12:0/3793744771 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c40074e0 con 0x7fc0dc0d3270 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.593+0000 7fc0d3fff640 1 -- 172.21.15.12:0/3793744771 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0c4007680 con 0x7fc0dc0d3270 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.593+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0dc154e30 con 0x7fc0dc0d3270 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.593+0000 7fc0d3fff640 1 -- 172.21.15.12:0/3793744771 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4004a00 con 0x7fc0dc0d3270 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.593+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0dc161b90 con 0x7fc0dc0d3270 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.593+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0dc0689e0 con 0x7fc0dc0d3270 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.594+0000 7fc0d3fff640 1 -- 172.21.15.12:0/3793744771 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0c401a020 con 0x7fc0dc0d3270 2024-04-15T15:22:48.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.594+0000 7fc0d3fff640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.594+0000 7fc0d3fff640 1 -- 172.21.15.12:0/3793744771 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc0c408b520 con 0x7fc0dc0d3270 2024-04-15T15:22:48.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.595+0000 7fc0daffd640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.595+0000 7fc0daffd640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009ac0 tx=0x7fc0d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.599+0000 7fc0d3fff640 1 -- 172.21.15.12:0/3793744771 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0c4059720 con 0x7fc0dc0d3270 2024-04-15T15:22:48.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.784+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0dc0ba500 con 0x7fc0dc0d3270 2024-04-15T15:22:48.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.784+0000 7fc0d3fff640 1 -- 172.21.15.12:0/3793744771 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7fc0c4059020 con 0x7fc0dc0d3270 2024-04-15T15:22:48.787 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:48.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0bc06c8f0 msgr2=0x7fc0bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009ac0 tx=0x7fc0d4008040 comp rx=0 tx=0).stop 2024-04-15T15:22:48.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc0d3270 msgr2=0x7fc0dc15ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc0d3270 0x7fc0dc15ec30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400c3c0 tx=0x7fc0c400c880 comp rx=0 tx=0).stop 2024-04-15T15:22:48.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 shutdown_connections 2024-04-15T15:22:48.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0dc15e700 0x7fc0dc0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0dc0d3270 0x7fc0dc15ec30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 --2- 172.21.15.12:0/3793744771 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc0dc0d2360 0x7fc0dc1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.786+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 >> 172.21.15.12:0/3793744771 conn(0x7fc0dc08bf60 msgr2=0x7fc0dc0d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:48.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.787+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 shutdown_connections 2024-04-15T15:22:48.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.787+0000 7fc0e1950640 1 -- 172.21.15.12:0/3793744771 wait complete. 2024-04-15T15:22:48.800 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.905+0000 7f0fdd6c5640 1 Processor -- start 2024-04-15T15:22:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.905+0000 7f0fdd6c5640 1 -- start start 2024-04-15T15:22:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.905+0000 7f0fdd6c5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fd80d3240 0x7f0fd80d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.905+0000 7f0fdd6c5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0fd80d1440 0x7f0fd80b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.905+0000 7f0fdd6c5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd80d2480 con 0x7f0fd815e790 2024-04-15T15:22:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.905+0000 7f0fdd6c5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0fd80d2db0 con 0x7f0fd80d3240 2024-04-15T15:22:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.905+0000 7f0fdd6c5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0fd80d2760 con 0x7f0fd80d1440 2024-04-15T15:22:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd67fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fd80d3240 0x7f0fd80d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd67fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0fd80d3240 0x7f0fd80d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49592/0 (socket says 172.21.15.12:49592) 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd67fc640 1 -- 172.21.15.12:0/1266241537 learned_addr learned my addr 172.21.15.12:0/1266241537 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd6ffd640 1 --1- 172.21.15.12:0/1266241537 >> v1:172.21.15.12:6789/0 conn(0x7f0fd815e790 0x7f0fd80cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42766/0 (socket says 172.21.15.12:42766) 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd67fc640 1 -- 172.21.15.12:0/1266241537 >> v2:172.21.15.55:3300/0 conn(0x7f0fd80d1440 msgr2=0x7f0fd80b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd67fc640 1 --2- 172.21.15.12:0/1266241537 >> v2:172.21.15.55:3300/0 conn(0x7f0fd80d1440 0x7f0fd80b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd67fc640 1 -- 172.21.15.12:0/1266241537 >> v1:172.21.15.12:6789/0 conn(0x7f0fd815e790 legacy=0x7f0fd80cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd67fc640 1 -- 172.21.15.12:0/1266241537 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd80d2a40 con 0x7f0fd80d3240 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.906+0000 7f0fd5ffb640 1 -- 172.21.15.12:0/1266241537 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134448829 0 0) 0x7f0fd80d2480 con 0x7f0fd815e790 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fd67fc640 1 --2- 172.21.15.12:0/1266241537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd80d0f00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80c8120 tx=0x7f0fcc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2877f3ac945e622c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fd5ffb640 1 -- 172.21.15.12:0/1266241537 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc034070 con 0x7f0fd80d3240 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fd5ffb640 1 -- 172.21.15.12:0/1266241537 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fcc030860 con 0x7f0fd80d3240 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fd5ffb640 1 -- 172.21.15.12:0/1266241537 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc030c30 con 0x7f0fd80d3240 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/1266241537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 msgr2=0x7f0fd80d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/1266241537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd80d0f00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80c8120 tx=0x7f0fcc02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/1266241537 shutdown_connections 2024-04-15T15:22:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/1266241537 >> v2:172.21.15.55:3300/0 conn(0x7f0fd80d1440 0x7f0fd80b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/1266241537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd80d0f00 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/1266241537 >> 172.21.15.12:0/1266241537 conn(0x7f0fd808bf60 msgr2=0x7f0fd80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/1266241537 shutdown_connections 2024-04-15T15:22:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.907+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/1266241537 wait complete. 2024-04-15T15:22:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fdd6c5640 1 Processor -- start 2024-04-15T15:22:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fdd6c5640 1 -- start start 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fdd6c5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fd80d1440 0x7f0fd81527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fdd6c5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd815c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fdd6c5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fd815e790 0x7f0fd80c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fdd6c5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd80d5180 con 0x7f0fd80d3240 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fdd6c5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd80d0430 con 0x7f0fd80d1440 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fdd6c5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd80d5300 con 0x7f0fd815e790 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fd67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fd67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd815c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49602/0 (socket says 172.21.15.12:49602) 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.909+0000 7f0fd67fc640 1 -- 172.21.15.12:0/493836519 learned_addr learned my addr 172.21.15.12:0/493836519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd77fe640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fd815e790 0x7f0fd80c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd6ffd640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fd80d1440 0x7f0fd81527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd67fc640 1 -- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fd815e790 msgr2=0x7f0fd80c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd67fc640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fd815e790 0x7f0fd80c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd67fc640 1 -- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fd80d1440 msgr2=0x7f0fd81527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd67fc640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fd80d1440 0x7f0fd81527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd67fc640 1 -- 172.21.15.12:0/493836519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd815a220 con 0x7f0fd80d3240 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd77fe640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fd815e790 0x7f0fd80c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd6ffd640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fd80d1440 0x7f0fd81527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fd67fc640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd815c080 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc02b9e0 tx=0x7f0fcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fbf7fe640 1 -- 172.21.15.12:0/493836519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc048070 con 0x7f0fd80d3240 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fbf7fe640 1 -- 172.21.15.12:0/493836519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fcc030660 con 0x7f0fd80d3240 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd815a500 con 0x7f0fd80d3240 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fbf7fe640 1 -- 172.21.15.12:0/493836519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc034040 con 0x7f0fd80d3240 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.910+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd815aac0 con 0x7f0fd80d3240 2024-04-15T15:22:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.911+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd80689e0 con 0x7f0fd80d3240 2024-04-15T15:22:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.912+0000 7f0fbf7fe640 1 -- 172.21.15.12:0/493836519 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0fcc04b070 con 0x7f0fd80d3240 2024-04-15T15:22:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.912+0000 7f0fbf7fe640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0fc406c8f0 0x7f0fc406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.912+0000 7f0fbf7fe640 1 -- 172.21.15.12:0/493836519 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f0fcc0adea0 con 0x7f0fd80d3240 2024-04-15T15:22:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.912+0000 7f0fd6ffd640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0fc406c8f0 0x7f0fc406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.913+0000 7f0fd6ffd640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0fc406c8f0 0x7f0fc406edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8000f60 tx=0x7f0fc8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.917+0000 7f0fbf7fe640 1 -- 172.21.15.12:0/493836519 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0fcc07c2f0 con 0x7f0fd80d3240 2024-04-15T15:22:48.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:48.942+0000 7f0fbf7fe640 1 -- 172.21.15.12:0/493836519 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0fcc037070 con 0x7f0fd80d3240 2024-04-15T15:22:49.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.102+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0fd80d2540 con 0x7f0fd80d3240 2024-04-15T15:22:49.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.102+0000 7f0fbf7fe640 1 -- 172.21.15.12:0/493836519 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f0fcc093d60 con 0x7f0fd80d3240 2024-04-15T15:22:49.105 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0fc406c8f0 msgr2=0x7f0fc406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0fc406c8f0 0x7f0fc406edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8000f60 tx=0x7f0fc8009040 comp rx=0 tx=0).stop 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 msgr2=0x7f0fd815c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd815c080 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc02b9e0 tx=0x7f0fcc004730 comp rx=0 tx=0).stop 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 shutdown_connections 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fd815e790 0x7f0fd80c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0fd80d3240 0x7f0fd815c080 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0fc406c8f0 0x7f0fc406edb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 --2- 172.21.15.12:0/493836519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fd80d1440 0x7f0fd81527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 >> 172.21.15.12:0/493836519 conn(0x7f0fd808bf60 msgr2=0x7f0fd80b9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 shutdown_connections 2024-04-15T15:22:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.104+0000 7f0fdd6c5640 1 -- 172.21.15.12:0/493836519 wait complete. 2024-04-15T15:22:49.117 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.223+0000 7fc6c635f640 1 Processor -- start 2024-04-15T15:22:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.223+0000 7fc6c635f640 1 -- start start 2024-04-15T15:22:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6c635f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6c0072d80 0x7fc6c0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6c635f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00d0da0 0x7fc6c00af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6c635f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6c015db40 con 0x7fc6c00af3e0 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6c635f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc6c015cfc0 con 0x7fc6c0072d80 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6c635f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c015d2a0 con 0x7fc6c00d0da0 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6bffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6c0072d80 0x7fc6c0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6bffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6c0072d80 0x7fc6c0071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49610/0 (socket says 172.21.15.12:49610) 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6bffff640 1 -- 172.21.15.12:0/4117408689 learned_addr learned my addr 172.21.15.12:0/4117408689 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.224+0000 7fc6c48fc640 1 --2- 172.21.15.12:0/4117408689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00d0da0 0x7fc6c00af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6bffff640 1 -- 172.21.15.12:0/4117408689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00d0da0 msgr2=0x7fc6c00af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6bffff640 1 --2- 172.21.15.12:0/4117408689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00d0da0 0x7fc6c00af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6bffff640 1 -- 172.21.15.12:0/4117408689 >> v1:172.21.15.55:6789/0 conn(0x7fc6c00af3e0 legacy=0x7fc6c00d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6bffff640 1 -- 172.21.15.12:0/4117408689 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c015d580 con 0x7fc6c0072d80 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6c48fc640 1 --2- 172.21.15.12:0/4117408689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00d0da0 0x7fc6c00af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6bffff640 1 --2- 172.21.15.12:0/4117408689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c0072d80 0x7fc6c0071d10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009870 tx=0x7fc6b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb7676e84f31bde server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6bf7fe640 1 -- 172.21.15.12:0/4117408689 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0035020 con 0x7fc6c0072d80 2024-04-15T15:22:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6bf7fe640 1 -- 172.21.15.12:0/4117408689 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b0031640 con 0x7fc6c0072d80 2024-04-15T15:22:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.225+0000 7fc6bf7fe640 1 -- 172.21.15.12:0/4117408689 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b002d040 con 0x7fc6c0072d80 2024-04-15T15:22:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.226+0000 7fc6c635f640 1 -- 172.21.15.12:0/4117408689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c0072d80 msgr2=0x7fc6c0071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.226+0000 7fc6c635f640 1 --2- 172.21.15.12:0/4117408689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c0072d80 0x7fc6c0071d10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009870 tx=0x7fc6b002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.226+0000 7fc6c635f640 1 -- 172.21.15.12:0/4117408689 shutdown_connections 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.226+0000 7fc6c635f640 1 --2- 172.21.15.12:0/4117408689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00d0da0 0x7fc6c00af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.226+0000 7fc6c635f640 1 --2- 172.21.15.12:0/4117408689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c0072d80 0x7fc6c0071d10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.226+0000 7fc6c635f640 1 -- 172.21.15.12:0/4117408689 >> 172.21.15.12:0/4117408689 conn(0x7fc6c006be00 msgr2=0x7fc6c006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.226+0000 7fc6c635f640 1 -- 172.21.15.12:0/4117408689 shutdown_connections 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.226+0000 7fc6c635f640 1 -- 172.21.15.12:0/4117408689 wait complete. 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.227+0000 7fc6c635f640 1 Processor -- start 2024-04-15T15:22:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.227+0000 7fc6c635f640 1 -- start start 2024-04-15T15:22:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c635f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6c0072d80 0x7fc6c0157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c635f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00af3e0 0x7fc6c014e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c635f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c00d0da0 0x7fc6c01550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c635f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c015cd10 con 0x7fc6c00d0da0 2024-04-15T15:22:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c635f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c015cb90 con 0x7fc6c0072d80 2024-04-15T15:22:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c635f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c006e060 con 0x7fc6c00af3e0 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c00d0da0 0x7fc6c01550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c00d0da0 0x7fc6c01550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49612/0 (socket says 172.21.15.12:49612) 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 -- 172.21.15.12:0/3015759551 learned_addr learned my addr 172.21.15.12:0/3015759551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6bffff640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6c0072d80 0x7fc6c0157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6b7fff640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00af3e0 0x7fc6c014e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 -- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00af3e0 msgr2=0x7fc6c014e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00af3e0 0x7fc6c014e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 -- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6c0072d80 msgr2=0x7fc6c0157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6c0072d80 0x7fc6c0157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 -- 172.21.15.12:0/3015759551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c015a530 con 0x7fc6c00d0da0 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.228+0000 7fc6c48fc640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c00d0da0 0x7fc6c01550d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8007dc0 tx=0x7fc6b800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.229+0000 7fc6bd7fa640 1 -- 172.21.15.12:0/3015759551 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b801a070 con 0x7fc6c00d0da0 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.229+0000 7fc6bd7fa640 1 -- 172.21.15.12:0/3015759551 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b8003990 con 0x7fc6c00d0da0 2024-04-15T15:22:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.229+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c014f370 con 0x7fc6c00d0da0 2024-04-15T15:22:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.229+0000 7fc6bd7fa640 1 -- 172.21.15.12:0/3015759551 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8003d80 con 0x7fc6c00d0da0 2024-04-15T15:22:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.229+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c014f930 con 0x7fc6c00d0da0 2024-04-15T15:22:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.230+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c00689e0 con 0x7fc6c00d0da0 2024-04-15T15:22:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.230+0000 7fc6bd7fa640 1 -- 172.21.15.12:0/3015759551 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc6b801d070 con 0x7fc6c00d0da0 2024-04-15T15:22:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.231+0000 7fc6bd7fa640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc6a006c980 0x7fc6a006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.231+0000 7fc6bd7fa640 1 -- 172.21.15.12:0/3015759551 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc6b808c0f0 con 0x7fc6c00d0da0 2024-04-15T15:22:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.231+0000 7fc6bffff640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc6a006c980 0x7fc6a006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.231+0000 7fc6bffff640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc6a006c980 0x7fc6a006ee40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0036000 tx=0x7fc6b003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.235+0000 7fc6bd7fa640 1 -- 172.21.15.12:0/3015759551 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6b8010070 con 0x7fc6c00d0da0 2024-04-15T15:22:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.422+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc6c0072090 con 0x7fc6c00d0da0 2024-04-15T15:22:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.423+0000 7fc6bd7fa640 1 -- 172.21.15.12:0/3015759551 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7fc6b805a290 con 0x7fc6c00d0da0 2024-04-15T15:22:49.425 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:49.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.424+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc6a006c980 msgr2=0x7fc6a006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.424+0000 7fc6c635f640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc6a006c980 0x7fc6a006ee40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0036000 tx=0x7fc6b003c040 comp rx=0 tx=0).stop 2024-04-15T15:22:49.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.424+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c00d0da0 msgr2=0x7fc6c01550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.424+0000 7fc6c635f640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c00d0da0 0x7fc6c01550d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8007dc0 tx=0x7fc6b800d4a0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.424+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 shutdown_connections 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.424+0000 7fc6c635f640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6c00d0da0 0x7fc6c01550d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.425+0000 7fc6c635f640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc6c00af3e0 0x7fc6c014e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.425+0000 7fc6c635f640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc6a006c980 0x7fc6a006ee40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.425+0000 7fc6c635f640 1 --2- 172.21.15.12:0/3015759551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6c0072d80 0x7fc6c0157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.425+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 >> 172.21.15.12:0/3015759551 conn(0x7fc6c006be00 msgr2=0x7fc6c00b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.425+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 shutdown_connections 2024-04-15T15:22:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.425+0000 7fc6c635f640 1 -- 172.21.15.12:0/3015759551 wait complete. 2024-04-15T15:22:49.438 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.541+0000 7f73ac1d4640 1 Processor -- start 2024-04-15T15:22:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.541+0000 7f73ac1d4640 1 -- start start 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73ac1d4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f73a4160420 0x7f73a415fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73ac1d4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a4070720 con 0x7f73a409a9d0 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73ac1d4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a4070a90 con 0x7f73a40701f0 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73ac1d4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f73a409ba90 con 0x7f73a4160420 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73a9f70640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f73a409a9d0 0x7f73a409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42778/0 (socket says 172.21.15.12:42778) 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73a9f70640 1 -- 172.21.15.12:0/3920731666 learned_addr learned my addr 172.21.15.12:0/3920731666 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73aa771640 1 --2- 172.21.15.12:0/3920731666 >> v2:172.21.15.12:3300/0 conn(0x7f73a4160420 0x7f73a415fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73aa771640 1 -- 172.21.15.12:0/3920731666 >> v1:172.21.15.55:6789/0 conn(0x7f73a40701f0 legacy=0x7f73a40d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73aa771640 1 -- 172.21.15.12:0/3920731666 >> v1:172.21.15.12:6789/0 conn(0x7f73a409a9d0 legacy=0x7f73a409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73a8f6e640 1 -- 172.21.15.12:0/3920731666 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253483475 0 0) 0x7f73a4070720 con 0x7f73a409a9d0 2024-04-15T15:22:49.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.542+0000 7f73aa771640 1 -- 172.21.15.12:0/3920731666 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a4070d70 con 0x7f73a4160420 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.543+0000 7f73aa771640 1 --2- 172.21.15.12:0/3920731666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a4160420 0x7f73a415fb30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f739c0097a0 tx=0x7f739c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f44d82ea488a771 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.543+0000 7f73a8f6e640 1 -- 172.21.15.12:0/3920731666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c034070 con 0x7f73a4160420 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.543+0000 7f73a8f6e640 1 -- 172.21.15.12:0/3920731666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f739c030610 con 0x7f73a4160420 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.543+0000 7f73a8f6e640 1 -- 172.21.15.12:0/3920731666 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c0309e0 con 0x7f73a4160420 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.544+0000 7f73ac1d4640 1 -- 172.21.15.12:0/3920731666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a4160420 msgr2=0x7f73a415fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.544+0000 7f73ac1d4640 1 --2- 172.21.15.12:0/3920731666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a4160420 0x7f73a415fb30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f739c0097a0 tx=0x7f739c02b960 comp rx=0 tx=0).stop 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.544+0000 7f73ac1d4640 1 -- 172.21.15.12:0/3920731666 shutdown_connections 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.544+0000 7f73ac1d4640 1 --2- 172.21.15.12:0/3920731666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a4160420 0x7f73a415fb30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.544+0000 7f73ac1d4640 1 -- 172.21.15.12:0/3920731666 >> 172.21.15.12:0/3920731666 conn(0x7f73a40d2460 msgr2=0x7f73a40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.544+0000 7f73ac1d4640 1 -- 172.21.15.12:0/3920731666 shutdown_connections 2024-04-15T15:22:49.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.544+0000 7f73ac1d4640 1 -- 172.21.15.12:0/3920731666 wait complete. 2024-04-15T15:22:49.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.545+0000 7f73ac1d4640 1 Processor -- start 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.545+0000 7f73ac1d4640 1 -- start start 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73ac1d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73a40701f0 0x7f73a4153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73ac1d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a409a9d0 0x7f73a40b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73ac1d4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73a4160420 0x7f73a40cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73ac1d4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f73a40d08c0 con 0x7f73a409a9d0 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73ac1d4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f73a4160070 con 0x7f73a40701f0 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73ac1d4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f73a40b1c60 con 0x7f73a4160420 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a409a9d0 0x7f73a40b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a409a9d0 0x7f73a40b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49636/0 (socket says 172.21.15.12:49636) 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 -- 172.21.15.12:0/1371198910 learned_addr learned my addr 172.21.15.12:0/1371198910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a9f70640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73a40701f0 0x7f73a4153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73aa771640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73a4160420 0x7f73a40cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 -- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73a4160420 msgr2=0x7f73a40cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73a4160420 0x7f73a40cf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 -- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73a40701f0 msgr2=0x7f73a4153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73a40701f0 0x7f73a4153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 -- 172.21.15.12:0/1371198910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a4157a00 con 0x7f73a409a9d0 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.546+0000 7f73a976f640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a409a9d0 0x7f73a40b6480 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f739800c3f0 tx=0x7f739800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.547+0000 7f73a27fc640 1 -- 172.21.15.12:0/1371198910 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398017070 con 0x7f73a409a9d0 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.547+0000 7f73a27fc640 1 -- 172.21.15.12:0/1371198910 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73980052f0 con 0x7f73a409a9d0 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.547+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a40b7950 con 0x7f73a409a9d0 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.547+0000 7f73a27fc640 1 -- 172.21.15.12:0/1371198910 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398003830 con 0x7f73a409a9d0 2024-04-15T15:22:49.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.547+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a40b7f10 con 0x7f73a409a9d0 2024-04-15T15:22:49.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.548+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a40b8150 con 0x7f73a409a9d0 2024-04-15T15:22:49.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.548+0000 7f73a27fc640 1 -- 172.21.15.12:0/1371198910 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f739801a070 con 0x7f73a409a9d0 2024-04-15T15:22:49.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.548+0000 7f73a27fc640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f739006c9b0 0x7f739006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.549+0000 7f73a27fc640 1 -- 172.21.15.12:0/1371198910 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f739808afa0 con 0x7f73a409a9d0 2024-04-15T15:22:49.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.549+0000 7f73a9f70640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f739006c9b0 0x7f739006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.549+0000 7f73a9f70640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f739006c9b0 0x7f739006ee70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f73940012e0 tx=0x7f7394004d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.553+0000 7f73a27fc640 1 -- 172.21.15.12:0/1371198910 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7398058c00 con 0x7f73a409a9d0 2024-04-15T15:22:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.749+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f73a40689e0 con 0x7f73a409a9d0 2024-04-15T15:22:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.749+0000 7f73a27fc640 1 -- 172.21.15.12:0/1371198910 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7f739805cf00 con 0x7f73a409a9d0 2024-04-15T15:22:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f739006c9b0 msgr2=0x7f739006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f739006c9b0 0x7f739006ee70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f73940012e0 tx=0x7f7394004d40 comp rx=0 tx=0).stop 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a409a9d0 msgr2=0x7f73a40b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a409a9d0 0x7f73a40b6480 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f739800c3f0 tx=0x7f739800c8b0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 shutdown_connections 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73a4160420 0x7f73a40cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73a409a9d0 0x7f73a40b6480 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f739006c9b0 0x7f739006ee70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 --2- 172.21.15.12:0/1371198910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73a40701f0 0x7f73a4153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 >> 172.21.15.12:0/1371198910 conn(0x7f73a40d2460 msgr2=0x7f73a40b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 shutdown_connections 2024-04-15T15:22:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.751+0000 7f73ac1d4640 1 -- 172.21.15.12:0/1371198910 wait complete. 2024-04-15T15:22:49.764 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:22:49.764 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:49.780 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:49.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.922+0000 7f74af78d640 1 Processor -- start 2024-04-15T15:22:49.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.922+0000 7f74af78d640 1 -- start start 2024-04-15T15:22:49.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.922+0000 7f74af78d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f74a8158770 0x7f74a815bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74af78d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a80b5230 0x7f74a8072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74af78d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74a8072c60 0x7f74a815e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74af78d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f74a80b4da0 con 0x7f74a8158770 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74af78d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f74a815c1d0 con 0x7f74a8072c60 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74af78d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f74a80b4190 con 0x7f74a80b5230 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74ad529640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f74a8158770 0x7f74a815bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74ad529640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f74a8158770 0x7f74a815bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49638/0 (socket says 172.21.15.12:49638) 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74ad529640 1 -- 172.21.15.12:0/3064987934 learned_addr learned my addr 172.21.15.12:0/3064987934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74add2a640 1 --2- 172.21.15.12:0/3064987934 >> v2:172.21.15.55:3300/0 conn(0x7f74a8072c60 0x7f74a815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74acd28640 1 --2- 172.21.15.12:0/3064987934 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a80b5230 0x7f74a8072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74ad529640 1 -- 172.21.15.12:0/3064987934 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a80b5230 msgr2=0x7f74a8072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:49.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74ad529640 1 --2- 172.21.15.12:0/3064987934 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a80b5230 0x7f74a8072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74ad529640 1 -- 172.21.15.12:0/3064987934 >> v2:172.21.15.55:3300/0 conn(0x7f74a8072c60 msgr2=0x7f74a815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:49.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74ad529640 1 --2- 172.21.15.12:0/3064987934 >> v2:172.21.15.55:3300/0 conn(0x7f74a8072c60 0x7f74a815e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74ad529640 1 -- 172.21.15.12:0/3064987934 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a80b4470 con 0x7f74a8158770 2024-04-15T15:22:49.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.923+0000 7f74acd28640 1 --2- 172.21.15.12:0/3064987934 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a80b5230 0x7f74a8072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:49.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.924+0000 7f74ad529640 1 --2- 172.21.15.12:0/3064987934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a8158770 0x7f74a815bc90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f74a80cc540 tx=0x7f749c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fced208e744aa10 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.924+0000 7f749bfff640 1 -- 172.21.15.12:0/3064987934 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c03b070 con 0x7f74a8158770 2024-04-15T15:22:49.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.924+0000 7f749bfff640 1 -- 172.21.15.12:0/3064987934 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f749c036440 con 0x7f74a8158770 2024-04-15T15:22:49.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.924+0000 7f749bfff640 1 -- 172.21.15.12:0/3064987934 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c036810 con 0x7f74a8158770 2024-04-15T15:22:49.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 -- 172.21.15.12:0/3064987934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a8158770 msgr2=0x7f74a815bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 --2- 172.21.15.12:0/3064987934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a8158770 0x7f74a815bc90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f74a80cc540 tx=0x7f749c02e0e0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 -- 172.21.15.12:0/3064987934 shutdown_connections 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 --2- 172.21.15.12:0/3064987934 >> v2:172.21.15.55:3300/0 conn(0x7f74a8072c60 0x7f74a815e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 --2- 172.21.15.12:0/3064987934 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a80b5230 0x7f74a8072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 --2- 172.21.15.12:0/3064987934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a8158770 0x7f74a815bc90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 -- 172.21.15.12:0/3064987934 >> 172.21.15.12:0/3064987934 conn(0x7f74a806be00 msgr2=0x7f74a806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 -- 172.21.15.12:0/3064987934 shutdown_connections 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.925+0000 7f74af78d640 1 -- 172.21.15.12:0/3064987934 wait complete. 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.926+0000 7f74af78d640 1 Processor -- start 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.926+0000 7f74af78d640 1 -- start start 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74af78d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a8072c60 0x7f74a8151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74af78d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a80b5230 0x7f74a814fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74af78d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a8158770 0x7f74a81efd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74af78d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f74a8071fa0 con 0x7f74a80b5230 2024-04-15T15:22:49.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74af78d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f74a80b4a30 con 0x7f74a8158770 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74af78d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f74a8158b50 con 0x7f74a8072c60 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a80b5230 0x7f74a814fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a80b5230 0x7f74a814fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49652/0 (socket says 172.21.15.12:49652) 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 -- 172.21.15.12:0/2807412858 learned_addr learned my addr 172.21.15.12:0/2807412858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74ad529640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a8072c60 0x7f74a8151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 -- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a8072c60 msgr2=0x7f74a8151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a8072c60 0x7f74a8151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 -- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a8158770 msgr2=0x7f74a81efd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a8158770 0x7f74a81efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 -- 172.21.15.12:0/2807412858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a81f3cd0 con 0x7f74a80b5230 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74acd28640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a80b5230 0x7f74a814fbb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f74a400e810 tx=0x7f74a400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f7499ffb640 1 -- 172.21.15.12:0/2807412858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a400f040 con 0x7f74a80b5230 2024-04-15T15:22:49.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f7499ffb640 1 -- 172.21.15.12:0/2807412858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a4004030 con 0x7f74a80b5230 2024-04-15T15:22:49.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a81f3fb0 con 0x7f74a80b5230 2024-04-15T15:22:49.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f7499ffb640 1 -- 172.21.15.12:0/2807412858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a4004400 con 0x7f74a80b5230 2024-04-15T15:22:49.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.927+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a81f4570 con 0x7f74a80b5230 2024-04-15T15:22:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.928+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a80689e0 con 0x7f74a80b5230 2024-04-15T15:22:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.929+0000 7f7499ffb640 1 -- 172.21.15.12:0/2807412858 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f74a400c650 con 0x7f74a80b5230 2024-04-15T15:22:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.929+0000 7f7499ffb640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748c06c980 0x7f748c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.929+0000 7f7499ffb640 1 -- 172.21.15.12:0/2807412858 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f74a408ba70 con 0x7f74a80b5230 2024-04-15T15:22:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.930+0000 7f74ad529640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748c06c980 0x7f748c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.930+0000 7f74ad529640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748c06c980 0x7f748c06ee40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f749c038000 tx=0x7f749c032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:49.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:49.934+0000 7f7499ffb640 1 -- 172.21.15.12:0/2807412858 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74a4017020 con 0x7f74a80b5230 2024-04-15T15:22:50.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.118+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74a80b3790 con 0x7f74a80b5230 2024-04-15T15:22:50.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.119+0000 7f7499ffb640 1 -- 172.21.15.12:0/2807412858 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16906 (secure 0 0 0) 0x7f74a4059ca0 con 0x7f74a80b5230 2024-04-15T15:22:50.122 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:50.122 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":39,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:22:34.287725+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748c06c980 msgr2=0x7f748c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748c06c980 0x7f748c06ee40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f749c038000 tx=0x7f749c032000 comp rx=0 tx=0).stop 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a80b5230 msgr2=0x7f74a814fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a80b5230 0x7f74a814fbb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f74a400e810 tx=0x7f74a400ecd0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 shutdown_connections 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748c06c980 0x7f748c06ee40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a8158770 0x7f74a81efd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a80b5230 0x7f74a814fbb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 --2- 172.21.15.12:0/2807412858 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a8072c60 0x7f74a8151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 >> 172.21.15.12:0/2807412858 conn(0x7f74a806be00 msgr2=0x7f74a81594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 shutdown_connections 2024-04-15T15:22:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.121+0000 7f74af78d640 1 -- 172.21.15.12:0/2807412858 wait complete. 2024-04-15T15:22:50.134 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:22:50.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.241+0000 7f5174874640 1 Processor -- start 2024-04-15T15:22:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.241+0000 7f5174874640 1 -- start start 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5174874640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f51700ce8e0 0x7f5170070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5174874640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5170071d90 0x7f517009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5174874640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f517006ff30 con 0x7f5170071240 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5174874640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5170070820 con 0x7f5170071d90 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5174874640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f51700cecc0 con 0x7f51700ce8e0 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5167fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5170071d90 0x7f517009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f516ed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5170071240 0x7f5170160460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42782/0 (socket says 172.21.15.12:42782) 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f516ed9d640 1 -- 172.21.15.12:0/2477605080 learned_addr learned my addr 172.21.15.12:0/2477605080 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f516e59c640 1 --2- 172.21.15.12:0/2477605080 >> v2:172.21.15.55:3300/0 conn(0x7f51700ce8e0 0x7f5170070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5167fff640 1 -- 172.21.15.12:0/2477605080 >> v2:172.21.15.55:3300/0 conn(0x7f51700ce8e0 msgr2=0x7f5170070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5167fff640 1 --2- 172.21.15.12:0/2477605080 >> v2:172.21.15.55:3300/0 conn(0x7f51700ce8e0 0x7f5170070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f516dd9b640 1 -- 172.21.15.12:0/2477605080 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2094664361 0 0) 0x7f517006ff30 con 0x7f5170071240 2024-04-15T15:22:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5167fff640 1 -- 172.21.15.12:0/2477605080 >> v1:172.21.15.12:6789/0 conn(0x7f5170071240 legacy=0x7f5170160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.242+0000 7f5167fff640 1 -- 172.21.15.12:0/2477605080 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517009b9d0 con 0x7f5170071d90 2024-04-15T15:22:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.243+0000 7f5167fff640 1 --2- 172.21.15.12:0/2477605080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071d90 0x7f517009a9d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f51700732e0 tx=0x7f515802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246a6e55fd3a8f83 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.243+0000 7f516dd9b640 1 -- 172.21.15.12:0/2477605080 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51580315a0 con 0x7f5170071d90 2024-04-15T15:22:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.243+0000 7f516dd9b640 1 -- 172.21.15.12:0/2477605080 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5158037070 con 0x7f5170071d90 2024-04-15T15:22:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.243+0000 7f516dd9b640 1 -- 172.21.15.12:0/2477605080 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158031960 con 0x7f5170071d90 2024-04-15T15:22:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.244+0000 7f5174874640 1 -- 172.21.15.12:0/2477605080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071d90 msgr2=0x7f517009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.244+0000 7f5174874640 1 --2- 172.21.15.12:0/2477605080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071d90 0x7f517009a9d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f51700732e0 tx=0x7f515802b840 comp rx=0 tx=0).stop 2024-04-15T15:22:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.244+0000 7f5174874640 1 -- 172.21.15.12:0/2477605080 shutdown_connections 2024-04-15T15:22:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.244+0000 7f5174874640 1 --2- 172.21.15.12:0/2477605080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071d90 0x7f517009a9d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.244+0000 7f5174874640 1 --2- 172.21.15.12:0/2477605080 >> v2:172.21.15.55:3300/0 conn(0x7f51700ce8e0 0x7f5170070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.244+0000 7f5174874640 1 -- 172.21.15.12:0/2477605080 >> 172.21.15.12:0/2477605080 conn(0x7f51700b3f40 msgr2=0x7f51700b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.244+0000 7f5174874640 1 -- 172.21.15.12:0/2477605080 shutdown_connections 2024-04-15T15:22:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.244+0000 7f5174874640 1 -- 172.21.15.12:0/2477605080 wait complete. 2024-04-15T15:22:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.245+0000 7f5174874640 1 Processor -- start 2024-04-15T15:22:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.245+0000 7f5174874640 1 -- start start 2024-04-15T15:22:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f5174874640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071240 0x7f51700b6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f5174874640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5170071d90 0x7f51701e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f5174874640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51700ce8e0 0x7f51700caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f5174874640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51700704b0 con 0x7f5170071240 2024-04-15T15:22:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f5174874640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51700701d0 con 0x7f5170071d90 2024-04-15T15:22:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f5174874640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5170071970 con 0x7f51700ce8e0 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071240 0x7f51700b6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071240 0x7f51700b6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49674/0 (socket says 172.21.15.12:49674) 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516e59c640 1 -- 172.21.15.12:0/503027496 learned_addr learned my addr 172.21.15.12:0/503027496 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f5167fff640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5170071d90 0x7f51701e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516ed9d640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51700ce8e0 0x7f51700caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516e59c640 1 -- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51700ce8e0 msgr2=0x7f51700caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516e59c640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51700ce8e0 0x7f51700caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516e59c640 1 -- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5170071d90 msgr2=0x7f51701e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516e59c640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5170071d90 0x7f51701e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.246+0000 7f516e59c640 1 -- 172.21.15.12:0/503027496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51700b5b50 con 0x7f5170071240 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.247+0000 7f516e59c640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071240 0x7f51700b6bf0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f516000c820 tx=0x7f516000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.247+0000 7f5166ffd640 1 -- 172.21.15.12:0/503027496 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160015070 con 0x7f5170071240 2024-04-15T15:22:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.247+0000 7f5166ffd640 1 -- 172.21.15.12:0/503027496 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5160004240 con 0x7f5170071240 2024-04-15T15:22:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.247+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51700b5e30 con 0x7f5170071240 2024-04-15T15:22:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.247+0000 7f5166ffd640 1 -- 172.21.15.12:0/503027496 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516000ce10 con 0x7f5170071240 2024-04-15T15:22:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.247+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51700b63f0 con 0x7f5170071240 2024-04-15T15:22:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.248+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51700689e0 con 0x7f5170071240 2024-04-15T15:22:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.248+0000 7f5166ffd640 1 -- 172.21.15.12:0/503027496 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f5160007830 con 0x7f5170071240 2024-04-15T15:22:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.249+0000 7f5166ffd640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f515006c980 0x7f515006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.249+0000 7f5166ffd640 1 -- 172.21.15.12:0/503027496 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f516008bd40 con 0x7f5170071240 2024-04-15T15:22:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.249+0000 7f5167fff640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f515006c980 0x7f515006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.249+0000 7f5167fff640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f515006c980 0x7f515006ee40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5158036000 tx=0x7f515803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.253+0000 7f5166ffd640 1 -- 172.21.15.12:0/503027496 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5160059ee0 con 0x7f5170071240 2024-04-15T15:22:50.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.439+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f51700b4ca0 con 0x7f5170071240 2024-04-15T15:22:50.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.440+0000 7f5166ffd640 1 -- 172.21.15.12:0/503027496 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f51600597e0 con 0x7f5170071240 2024-04-15T15:22:50.442 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f515006c980 msgr2=0x7f515006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f515006c980 0x7f515006ee40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5158036000 tx=0x7f515803d040 comp rx=0 tx=0).stop 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071240 msgr2=0x7f51700b6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071240 0x7f51700b6bf0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f516000c820 tx=0x7f516000cce0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 shutdown_connections 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51700ce8e0 0x7f51700caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f515006c980 0x7f515006ee40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5170071d90 0x7f51701e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 --2- 172.21.15.12:0/503027496 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5170071240 0x7f51700b6bf0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 >> 172.21.15.12:0/503027496 conn(0x7f51700b3f40 msgr2=0x7f51701e4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 shutdown_connections 2024-04-15T15:22:50.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.442+0000 7f5174874640 1 -- 172.21.15.12:0/503027496 wait complete. 2024-04-15T15:22:50.455 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:50.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.560+0000 7f0fa9b75640 1 Processor -- start 2024-04-15T15:22:50.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.560+0000 7f0fa9b75640 1 -- start start 2024-04-15T15:22:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa9b75640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415e790 0x7f0fa40b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa9b75640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa415eb70 con 0x7f0fa40b7be0 2024-04-15T15:22:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa9b75640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa40d3c80 con 0x7f0fa40d1cf0 2024-04-15T15:22:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa9b75640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa40d3790 con 0x7f0fa415e790 2024-04-15T15:22:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa2ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0fa40b7be0 0x7f0fa40d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42798/0 (socket says 172.21.15.12:42798) 2024-04-15T15:22:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa2ffd640 1 -- 172.21.15.12:0/1002750154 learned_addr learned my addr 172.21.15.12:0/1002750154 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:50.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa37fe640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415e790 0x7f0fa40b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 829610076 0 0) 0x7f0fa415eb70 con 0x7f0fa40b7be0 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.561+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f8c003520 con 0x7f0fa40b7be0 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1788493707 0 0) 0x7f0f8c003520 con 0x7f0fa40b7be0 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415e790 msgr2=0x7f0fa40b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415e790 0x7f0fa40b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 >> v1:172.21.15.55:6789/0 conn(0x7f0fa40d1cf0 legacy=0x7f0fa40d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa40d3f60 con 0x7f0fa40b7be0 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f0f98001690 con 0x7f0fa40b7be0 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 >> v1:172.21.15.12:6789/0 conn(0x7f0fa40b7be0 legacy=0x7f0fa40d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0039e0 0x7f0f8c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0f8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415e790 0x7f0f8c005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa40d3790 con 0x7f0f8c004800 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c00bb80 con 0x7f0f8c0039e0 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0f8c00bd80 con 0x7f0fa415e790 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa2ffd640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0f8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa37fe640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0039e0 0x7f0f8c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa3fff640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415e790 0x7f0f8c005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa2ffd640 1 -- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415e790 msgr2=0x7f0f8c005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa2ffd640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415e790 0x7f0f8c005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa2ffd640 1 -- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0039e0 msgr2=0x7f0f8c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa2ffd640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0039e0 0x7f0f8c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.562+0000 7f0fa2ffd640 1 -- 172.21.15.12:0/1002750154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8c003520 con 0x7f0f8c004800 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa2ffd640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0f8c004c80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0f9800c210 tx=0x7f0f8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17629f2db9f6e6f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f98007d10 con 0x7f0f8c004800 2024-04-15T15:22:50.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f98030070 con 0x7f0f8c004800 2024-04-15T15:22:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa27fc640 1 -- 172.21.15.12:0/1002750154 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f98005d30 con 0x7f0f8c004800 2024-04-15T15:22:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa9b75640 1 -- 172.21.15.12:0/1002750154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 msgr2=0x7f0f8c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa9b75640 1 --2- 172.21.15.12:0/1002750154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0f8c004c80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0f9800c210 tx=0x7f0f8c005830 comp rx=0 tx=0).stop 2024-04-15T15:22:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa3fff640 1 -- 172.21.15.12:0/1002750154 reap_dead start 2024-04-15T15:22:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa9b75640 1 -- 172.21.15.12:0/1002750154 shutdown_connections 2024-04-15T15:22:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa9b75640 1 -- 172.21.15.12:0/1002750154 >> 172.21.15.12:0/1002750154 conn(0x7f0fa4067b40 msgr2=0x7f0fa4067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa9b75640 1 -- 172.21.15.12:0/1002750154 shutdown_connections 2024-04-15T15:22:50.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.563+0000 7f0fa9b75640 1 -- 172.21.15.12:0/1002750154 wait complete. 2024-04-15T15:22:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.565+0000 7f0fa9b75640 1 Processor -- start 2024-04-15T15:22:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.565+0000 7f0fa9b75640 1 -- start start 2024-04-15T15:22:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.565+0000 7f0fa9b75640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0fa40cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa9b75640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fa4159760 0x7f0fa414ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa9b75640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415ae90 0x7f0fa415b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa9b75640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa40d5150 con 0x7f0f8c004800 2024-04-15T15:22:50.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa9b75640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa40d45b0 con 0x7f0fa4159760 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa9b75640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa40d52d0 con 0x7f0fa415ae90 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0fa40cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0fa40cae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49688/0 (socket says 172.21.15.12:49688) 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 -- 172.21.15.12:0/845767734 learned_addr learned my addr 172.21.15.12:0/845767734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa2ffd640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fa4159760 0x7f0fa414ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa3fff640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415ae90 0x7f0fa415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 -- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415ae90 msgr2=0x7f0fa415b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415ae90 0x7f0fa415b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 -- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fa4159760 msgr2=0x7f0fa414ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fa4159760 0x7f0fa414ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 -- 172.21.15.12:0/845767734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa415bc40 con 0x7f0f8c004800 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa37fe640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0fa40cae70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0f94011a00 tx=0x7f0f94011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0f8bfff640 1 -- 172.21.15.12:0/845767734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9400fb50 con 0x7f0f8c004800 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0f8bfff640 1 -- 172.21.15.12:0/845767734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f9400fcf0 con 0x7f0f8c004800 2024-04-15T15:22:50.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa415bf20 con 0x7f0f8c004800 2024-04-15T15:22:50.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0f8bfff640 1 -- 172.21.15.12:0/845767734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9400fea0 con 0x7f0f8c004800 2024-04-15T15:22:50.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.566+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa415c4e0 con 0x7f0f8c004800 2024-04-15T15:22:50.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.567+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fa40689e0 con 0x7f0f8c004800 2024-04-15T15:22:50.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.568+0000 7f0f8bfff640 1 -- 172.21.15.12:0/845767734 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0f94012040 con 0x7f0f8c004800 2024-04-15T15:22:50.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.568+0000 7f0f8bfff640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f9006c980 0x7f0f9006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.568+0000 7f0f8bfff640 1 -- 172.21.15.12:0/845767734 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f0f9408c3d0 con 0x7f0f8c004800 2024-04-15T15:22:50.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.568+0000 7f0fa2ffd640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f9006c980 0x7f0f9006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.569+0000 7f0fa2ffd640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f9006c980 0x7f0f9006ee40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0f98034000 tx=0x7f0f98007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.573+0000 7f0f8bfff640 1 -- 172.21.15.12:0/845767734 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f94019030 con 0x7f0f8c004800 2024-04-15T15:22:50.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.757+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0fa4161b60 con 0x7f0f8c004800 2024-04-15T15:22:50.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.758+0000 7f0f8bfff640 1 -- 172.21.15.12:0/845767734 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f0f9405a710 con 0x7f0f8c004800 2024-04-15T15:22:50.761 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:50.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f9006c980 msgr2=0x7f0f9006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f9006c980 0x7f0f9006ee40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0f98034000 tx=0x7f0f98007000 comp rx=0 tx=0).stop 2024-04-15T15:22:50.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 msgr2=0x7f0fa40cae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0fa40cae70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0f94011a00 tx=0x7f0f94011ec0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 shutdown_connections 2024-04-15T15:22:50.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0fa415ae90 0x7f0fa415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f9006c980 0x7f0f9006ee40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fa4159760 0x7f0fa414ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 --2- 172.21.15.12:0/845767734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c004800 0x7f0fa40cae70 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 >> 172.21.15.12:0/845767734 conn(0x7f0fa4067b40 msgr2=0x7f0fa40cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:50.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 shutdown_connections 2024-04-15T15:22:50.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.760+0000 7f0fa9b75640 1 -- 172.21.15.12:0/845767734 wait complete. 2024-04-15T15:22:50.773 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.878+0000 7efec4f67640 1 Processor -- start 2024-04-15T15:22:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.878+0000 7efec4f67640 1 -- start start 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efec4f67640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec015e790 0x7efec00b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efec4f67640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efec00b7be0 0x7efec00d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efec4f67640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec015eb70 con 0x7efec00d1b20 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efec4f67640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7efec00d3790 con 0x7efec00b7be0 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efec4f67640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7efec00d45b0 con 0x7efec015e790 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efeb5d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efec00b7be0 0x7efec00d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efebed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efec00d1b20 0x7efec00d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42802/0 (socket says 172.21.15.12:42802) 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efebed9d640 1 -- 172.21.15.12:0/1264337959 learned_addr learned my addr 172.21.15.12:0/1264337959 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efebe59c640 1 --2- 172.21.15.12:0/1264337959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec015e790 0x7efec00b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efeb5d9b640 1 -- 172.21.15.12:0/1264337959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec015e790 msgr2=0x7efec00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:50.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efeb5d9b640 1 --2- 172.21.15.12:0/1264337959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec015e790 0x7efec00b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efeb5d9b640 1 -- 172.21.15.12:0/1264337959 >> v1:172.21.15.12:6789/0 conn(0x7efec00d1b20 legacy=0x7efec00d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efeb5d9b640 1 -- 172.21.15.12:0/1264337959 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec00d3c80 con 0x7efec00b7be0 2024-04-15T15:22:50.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efebdd9b640 1 -- 172.21.15.12:0/1264337959 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066269318 0 0) 0x7efec015eb70 con 0x7efec00d1b20 2024-04-15T15:22:50.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.879+0000 7efebe59c640 1 --2- 172.21.15.12:0/1264337959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec015e790 0x7efec00b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.880+0000 7efeb5d9b640 1 --2- 172.21.15.12:0/1264337959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec00b7be0 0x7efec00d15e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7efea80097a0 tx=0x7efea802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad95aa53c37a401 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.880+0000 7efebdd9b640 1 -- 172.21.15.12:0/1264337959 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8039070 con 0x7efec00b7be0 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.880+0000 7efebdd9b640 1 -- 172.21.15.12:0/1264337959 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efea80316f0 con 0x7efec00b7be0 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.880+0000 7efebdd9b640 1 -- 172.21.15.12:0/1264337959 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8031ac0 con 0x7efec00b7be0 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.881+0000 7efec4f67640 1 -- 172.21.15.12:0/1264337959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec00b7be0 msgr2=0x7efec00d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.881+0000 7efec4f67640 1 --2- 172.21.15.12:0/1264337959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec00b7be0 0x7efec00d15e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7efea80097a0 tx=0x7efea802b790 comp rx=0 tx=0).stop 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.881+0000 7efec4f67640 1 -- 172.21.15.12:0/1264337959 shutdown_connections 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.881+0000 7efec4f67640 1 --2- 172.21.15.12:0/1264337959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec00b7be0 0x7efec00d15e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.881+0000 7efec4f67640 1 --2- 172.21.15.12:0/1264337959 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec015e790 0x7efec00b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.881+0000 7efec4f67640 1 -- 172.21.15.12:0/1264337959 >> 172.21.15.12:0/1264337959 conn(0x7efec0067b40 msgr2=0x7efec0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:50.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.881+0000 7efec4f67640 1 -- 172.21.15.12:0/1264337959 shutdown_connections 2024-04-15T15:22:50.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.881+0000 7efec4f67640 1 -- 172.21.15.12:0/1264337959 wait complete. 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.882+0000 7efec4f67640 1 Processor -- start 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.882+0000 7efec4f67640 1 -- start start 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efec4f67640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efec00b7be0 0x7efec00ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efec4f67640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec00d1b20 0x7efec00c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efec4f67640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec015e790 0x7efec014e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efec4f67640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efec00d5150 con 0x7efec015e790 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efec4f67640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7efec00d4240 con 0x7efec00b7be0 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efec4f67640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7efec00d52d0 con 0x7efec00d1b20 2024-04-15T15:22:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efebed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec015e790 0x7efec014e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efebed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec015e790 0x7efec014e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49714/0 (socket says 172.21.15.12:49714) 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efebed9d640 1 -- 172.21.15.12:0/2925721128 learned_addr learned my addr 172.21.15.12:0/2925721128 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efeb5d9b640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec00d1b20 0x7efec00c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efebe59c640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efec00b7be0 0x7efec00ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efebed9d640 1 -- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec00d1b20 msgr2=0x7efec00c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.883+0000 7efebed9d640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec00d1b20 0x7efec00c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efebed9d640 1 -- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efec00b7be0 msgr2=0x7efec00ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efebed9d640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efec00b7be0 0x7efec00ca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efebed9d640 1 -- 172.21.15.12:0/2925721128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec00c92d0 con 0x7efec015e790 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efebe59c640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efec00b7be0 0x7efec00ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efebed9d640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec015e790 0x7efec014e7b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7efeb800ca40 tx=0x7efeb800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efeb77fe640 1 -- 172.21.15.12:0/2925721128 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb800abd0 con 0x7efec015e790 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efeb77fe640 1 -- 172.21.15.12:0/2925721128 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb800adb0 con 0x7efec015e790 2024-04-15T15:22:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efeb77fe640 1 -- 172.21.15.12:0/2925721128 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb8012640 con 0x7efec015e790 2024-04-15T15:22:50.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec00c95b0 con 0x7efec015e790 2024-04-15T15:22:50.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.884+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec00c9b70 con 0x7efec015e790 2024-04-15T15:22:50.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.885+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec00689e0 con 0x7efec015e790 2024-04-15T15:22:50.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.885+0000 7efeb77fe640 1 -- 172.21.15.12:0/2925721128 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7efeb80127e0 con 0x7efec015e790 2024-04-15T15:22:50.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.886+0000 7efeb77fe640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7efea006c980 0x7efea006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:50.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.886+0000 7efeb77fe640 1 -- 172.21.15.12:0/2925721128 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7efeb808c2a0 con 0x7efec015e790 2024-04-15T15:22:50.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.886+0000 7efebe59c640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7efea006c980 0x7efea006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:50.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.886+0000 7efebe59c640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7efea006c980 0x7efea006ee40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efeb0006fd0 tx=0x7efeb0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:50.890+0000 7efeb77fe640 1 -- 172.21.15.12:0/2925721128 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efeb805a550 con 0x7efec015e790 2024-04-15T15:22:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.076+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efec00b5a40 con 0x7efec015e790 2024-04-15T15:22:51.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.076+0000 7efeb77fe640 1 -- 172.21.15.12:0/2925721128 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) ==== 100+0+11 (secure 0 0 0) 0x7efeb8059e50 con 0x7efec015e790 2024-04-15T15:22:51.079 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:51.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7efea006c980 msgr2=0x7efea006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7efea006c980 0x7efea006ee40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efeb0006fd0 tx=0x7efeb0008040 comp rx=0 tx=0).stop 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec015e790 msgr2=0x7efec014e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec015e790 0x7efec014e7b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7efeb800ca40 tx=0x7efeb800cf00 comp rx=0 tx=0).stop 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 shutdown_connections 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efec015e790 0x7efec014e7b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7efec00d1b20 0x7efec00c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7efea006c980 0x7efea006ee40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 --2- 172.21.15.12:0/2925721128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efec00b7be0 0x7efec00ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 >> 172.21.15.12:0/2925721128 conn(0x7efec0067b40 msgr2=0x7efec00d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 shutdown_connections 2024-04-15T15:22:51.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.079+0000 7efec4f67640 1 -- 172.21.15.12:0/2925721128 wait complete. 2024-04-15T15:22:51.093 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:51.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.197+0000 7f1f1c7ba640 1 Processor -- start 2024-04-15T15:22:51.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.197+0000 7f1f1c7ba640 1 -- start start 2024-04-15T15:22:51.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f1c7ba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1f1415e700 0x7f1f140b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:51.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f1c7ba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f140d2360 0x7f1f140d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:51.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f1c7ba640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f140b47a0 con 0x7f1f1415eae0 2024-04-15T15:22:51.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f1c7ba640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1f140b53b0 con 0x7f1f1415e700 2024-04-15T15:22:51.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f1c7ba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1f140b4a80 con 0x7f1f140d2360 2024-04-15T15:22:51.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f1ad57640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1f1415eae0 0x7f1f140cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42806/0 (socket says 172.21.15.12:42806) 2024-04-15T15:22:51.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f1ad57640 1 -- 172.21.15.12:0/2503283110 learned_addr learned my addr 172.21.15.12:0/2503283110 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:51.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f19d55640 1 --2- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f140d2360 0x7f1f140d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:51.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f1a556640 1 --2- 172.21.15.12:0/2503283110 >> v2:172.21.15.55:3300/0 conn(0x7f1f1415e700 0x7f1f140b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:51.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.198+0000 7f1f19554640 1 -- 172.21.15.12:0/2503283110 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280699984 0 0) 0x7f1f140b47a0 con 0x7f1f1415eae0 2024-04-15T15:22:51.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.199+0000 7f1f1a556640 1 -- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f140d2360 msgr2=0x7f1f140d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:51.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.199+0000 7f1f1a556640 1 --2- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f140d2360 0x7f1f140d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.199+0000 7f1f1a556640 1 -- 172.21.15.12:0/2503283110 >> v1:172.21.15.12:6789/0 conn(0x7f1f1415eae0 legacy=0x7f1f140cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:51.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.199+0000 7f1f1a556640 1 -- 172.21.15.12:0/2503283110 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f140b4d60 con 0x7f1f1415e700 2024-04-15T15:22:51.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.199+0000 7f1f19d55640 1 --2- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f140d2360 0x7f1f140d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:22:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.200+0000 7f1f1a556640 1 --2- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f1415e700 0x7f1f140b5840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009790 tx=0x7f1f0c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=465056766f78f07a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.200+0000 7f1f19554640 1 -- 172.21.15.12:0/2503283110 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c035020 con 0x7f1f1415e700 2024-04-15T15:22:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.200+0000 7f1f19554640 1 -- 172.21.15.12:0/2503283110 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f0c031800 con 0x7f1f1415e700 2024-04-15T15:22:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.200+0000 7f1f19554640 1 -- 172.21.15.12:0/2503283110 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c031bd0 con 0x7f1f1415e700 2024-04-15T15:22:51.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.200+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f1415e700 msgr2=0x7f1f140b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:51.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.200+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f1415e700 0x7f1f140b5840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009790 tx=0x7f1f0c02b8a0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.201+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/2503283110 shutdown_connections 2024-04-15T15:22:51.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.201+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f140d2360 0x7f1f140d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.201+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/2503283110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f1415e700 0x7f1f140b5840 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.201+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/2503283110 >> 172.21.15.12:0/2503283110 conn(0x7f1f1408bf60 msgr2=0x7f1f140d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.201+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/2503283110 shutdown_connections 2024-04-15T15:22:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.201+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/2503283110 wait complete. 2024-04-15T15:22:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.202+0000 7f1f1c7ba640 1 Processor -- start 2024-04-15T15:22:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.202+0000 7f1f1c7ba640 1 -- start start 2024-04-15T15:22:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.202+0000 7f1f1c7ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f140d2360 0x7f1f141f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:51.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1c7ba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f1415e700 0x7f1f140b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1c7ba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f1415eae0 0x7f1f141611d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1c7ba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1f140b5d80 con 0x7f1f1415eae0 2024-04-15T15:22:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1c7ba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1f140b4450 con 0x7f1f140d2360 2024-04-15T15:22:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1c7ba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1f140d1d40 con 0x7f1f1415e700 2024-04-15T15:22:51.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f1415eae0 0x7f1f141611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f1415eae0 0x7f1f141611d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49726/0 (socket says 172.21.15.12:49726) 2024-04-15T15:22:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 -- 172.21.15.12:0/3457703880 learned_addr learned my addr 172.21.15.12:0/3457703880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f19d55640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f1415e700 0x7f1f140b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1a556640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f140d2360 0x7f1f141f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 -- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f1415e700 msgr2=0x7f1f140b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:51.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f1415e700 0x7f1f140b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 -- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f140d2360 msgr2=0x7f1f141f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f140d2360 0x7f1f141f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 -- 172.21.15.12:0/3457703880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f141f47a0 con 0x7f1f1415eae0 2024-04-15T15:22:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1ad57640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f1415eae0 0x7f1f141611d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400c700 tx=0x7f1f0400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f12ffd640 1 -- 172.21.15.12:0/3457703880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04018070 con 0x7f1f1415eae0 2024-04-15T15:22:51.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f12ffd640 1 -- 172.21.15.12:0/3457703880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f0400ce10 con 0x7f1f1415eae0 2024-04-15T15:22:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f140c9580 con 0x7f1f1415eae0 2024-04-15T15:22:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f12ffd640 1 -- 172.21.15.12:0/3457703880 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0400a860 con 0x7f1f1415eae0 2024-04-15T15:22:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.203+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f140b9d90 con 0x7f1f1415eae0 2024-04-15T15:22:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.204+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f140689e0 con 0x7f1f1415eae0 2024-04-15T15:22:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.205+0000 7f1f12ffd640 1 -- 172.21.15.12:0/3457703880 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f1f0400aa00 con 0x7f1f1415eae0 2024-04-15T15:22:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.205+0000 7f1f12ffd640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f0006c980 0x7f1f0006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:51.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.206+0000 7f1f12ffd640 1 -- 172.21.15.12:0/3457703880 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f1f0408b2a0 con 0x7f1f1415eae0 2024-04-15T15:22:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.206+0000 7f1f1a556640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f0006c980 0x7f1f0006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.206+0000 7f1f1a556640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f0006c980 0x7f1f0006ee40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c036000 tx=0x7f1f0c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:51.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.210+0000 7f1f12ffd640 1 -- 172.21.15.12:0/3457703880 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f04058ef0 con 0x7f1f1415eae0 2024-04-15T15:22:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.408+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1f140ba500 con 0x7f1f1415eae0 2024-04-15T15:22:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.408+0000 7f1f12ffd640 1 -- 172.21.15.12:0/3457703880 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v40) ==== 80+0+0 (secure 0 0 0) 0x7f1f0405d1f0 con 0x7f1f1415eae0 2024-04-15T15:22:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f0006c980 msgr2=0x7f1f0006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f0006c980 0x7f1f0006ee40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c036000 tx=0x7f1f0c03d040 comp rx=0 tx=0).stop 2024-04-15T15:22:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f1415eae0 msgr2=0x7f1f141611d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f1415eae0 0x7f1f141611d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400c700 tx=0x7f1f0400cbc0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 shutdown_connections 2024-04-15T15:22:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f1415eae0 0x7f1f141611d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1f1415e700 0x7f1f140b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1f0006c980 0x7f1f0006ee40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 --2- 172.21.15.12:0/3457703880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f140d2360 0x7f1f141f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 >> 172.21.15.12:0/3457703880 conn(0x7f1f1408bf60 msgr2=0x7f1f1406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 shutdown_connections 2024-04-15T15:22:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:51.410+0000 7f1f1c7ba640 1 -- 172.21.15.12:0/3457703880 wait complete. 2024-04-15T15:22:51.423 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-04-15T15:22:51.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:51.424 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:22:51.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[54041]: fuse finished with error 0 and tester_r 0 2024-04-15T15:22:51.452 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:22:57.553 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:22:57.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:22:57.554 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:22:57.574 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:22:57.574 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:57.593 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:57.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.726+0000 7ff498a9a640 1 Processor -- start 2024-04-15T15:22:57.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.726+0000 7ff498a9a640 1 -- start start 2024-04-15T15:22:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.727+0000 7ff498a9a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff4941602d0 0x7ff4940cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.727+0000 7ff498a9a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 0x7ff4940d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.727+0000 7ff498a9a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4940cd6a0 con 0x7ff49415cbc0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.727+0000 7ff498a9a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff4940cd980 con 0x7ff4941602d0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.727+0000 7ff498a9a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff4940cdc60 con 0x7ff4941606b0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.727+0000 7ff492d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff49415cbc0 0x7ff49415d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47388/0 (socket says 172.21.15.12:47388) 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.727+0000 7ff492d9d640 1 -- 172.21.15.12:0/2679563674 learned_addr learned my addr 172.21.15.12:0/2679563674 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff49259c640 1 --2- 172.21.15.12:0/2679563674 >> v2:172.21.15.55:3300/0 conn(0x7ff4941602d0 0x7ff4940cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371938639 0 0) 0x7ff4940cd6a0 con 0x7ff49415cbc0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff48bfff640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 0x7ff4940d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff480003520 con 0x7ff49415cbc0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 734644020 0 0) 0x7ff480003520 con 0x7ff49415cbc0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 msgr2=0x7ff4940d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 0x7ff4940d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 >> v2:172.21.15.55:3300/0 conn(0x7ff4941602d0 msgr2=0x7ff4940cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 --2- 172.21.15.12:0/2679563674 >> v2:172.21.15.55:3300/0 conn(0x7ff4941602d0 0x7ff4940cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4940cdf40 con 0x7ff49415cbc0 2024-04-15T15:22:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7ff48c0019f0 con 0x7ff49415cbc0 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff49259c640 1 --2- 172.21.15.12:0/2679563674 >> v2:172.21.15.55:3300/0 conn(0x7ff4941602d0 0x7ff4940cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff48bfff640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 0x7ff4940d0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 >> v1:172.21.15.12:6789/0 conn(0x7ff49415cbc0 legacy=0x7ff49415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4800039b0 0x7ff480003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 0x7ff480004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff480004d40 0x7ff4800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff4940cd980 con 0x7ff480004d40 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff4940cdc60 con 0x7ff4800039b0 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff48000ba40 con 0x7ff4941606b0 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff49259c640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4800039b0 0x7ff480003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff492d9d640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff480004d40 0x7ff4800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.728+0000 7ff48bfff640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 0x7ff480004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff492d9d640 1 -- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 msgr2=0x7ff480004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:57.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff492d9d640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 0x7ff480004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff492d9d640 1 -- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4800039b0 msgr2=0x7ff480003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff492d9d640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4800039b0 0x7ff480003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff492d9d640 1 -- 172.21.15.12:0/2679563674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff480003520 con 0x7ff480004d40 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff48bfff640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff4941606b0 0x7ff480004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff492d9d640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff480004d40 0x7ff4800051c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00a8a0 tx=0x7ff480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c54dade9c87c02d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c00a4c0 con 0x7ff480004d40 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48c034650 con 0x7ff480004d40 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.729+0000 7ff491d9b640 1 -- 172.21.15.12:0/2679563674 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c034a40 con 0x7ff480004d40 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.730+0000 7ff498a9a640 1 -- 172.21.15.12:0/2679563674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff480004d40 msgr2=0x7ff4800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.730+0000 7ff498a9a640 1 --2- 172.21.15.12:0/2679563674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff480004d40 0x7ff4800051c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00a8a0 tx=0x7ff480005830 comp rx=0 tx=0).stop 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.730+0000 7ff492d9d640 1 -- 172.21.15.12:0/2679563674 reap_dead start 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.730+0000 7ff498a9a640 1 -- 172.21.15.12:0/2679563674 shutdown_connections 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.730+0000 7ff498a9a640 1 -- 172.21.15.12:0/2679563674 >> 172.21.15.12:0/2679563674 conn(0x7ff4940d2460 msgr2=0x7ff4940d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.730+0000 7ff498a9a640 1 -- 172.21.15.12:0/2679563674 shutdown_connections 2024-04-15T15:22:57.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.730+0000 7ff498a9a640 1 -- 172.21.15.12:0/2679563674 wait complete. 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.731+0000 7ff498a9a640 1 Processor -- start 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff498a9a640 1 -- start start 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff498a9a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff480004d40 0x7ff4941f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff498a9a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4940caa00 0x7ff4940b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff498a9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4940b0710 0x7ff4940b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff498a9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff49409bb50 con 0x7ff4940b0710 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff498a9a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff494160b10 con 0x7ff4940caa00 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff498a9a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7ff49409bcd0 con 0x7ff480004d40 2024-04-15T15:22:57.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff492d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4940b0710 0x7ff4940b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff492d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4940b0710 0x7ff4940b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41760/0 (socket says 172.21.15.12:41760) 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.732+0000 7ff492d9d640 1 -- 172.21.15.12:0/2030745852 learned_addr learned my addr 172.21.15.12:0/2030745852 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff49259c640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff480004d40 0x7ff4941f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff48bfff640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4940caa00 0x7ff4940b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff492d9d640 1 -- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff480004d40 msgr2=0x7ff4941f8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff492d9d640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff480004d40 0x7ff4941f8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff492d9d640 1 -- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4940caa00 msgr2=0x7ff4940b79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff492d9d640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4940caa00 0x7ff4940b79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff492d9d640 1 -- 172.21.15.12:0/2030745852 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4940b7f20 con 0x7ff4940b0710 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff492d9d640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4940b0710 0x7ff4940b0bc0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00a8a0 tx=0x7ff48c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff48affd640 1 -- 172.21.15.12:0/2030745852 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c002010 con 0x7ff4940b0710 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff48affd640 1 -- 172.21.15.12:0/2030745852 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48c044070 con 0x7ff4940b0710 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff48affd640 1 -- 172.21.15.12:0/2030745852 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c034c00 con 0x7ff4940b0710 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4940c5420 con 0x7ff4940b0710 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.733+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4940cb280 con 0x7ff4940b0710 2024-04-15T15:22:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.734+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4940689e0 con 0x7ff4940b0710 2024-04-15T15:22:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.735+0000 7ff48affd640 1 -- 172.21.15.12:0/2030745852 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff48c049460 con 0x7ff4940b0710 2024-04-15T15:22:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.735+0000 7ff48affd640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff47406c9b0 0x7ff47406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.736+0000 7ff48affd640 1 -- 172.21.15.12:0/2030745852 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7ff48c0ae250 con 0x7ff4940b0710 2024-04-15T15:22:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.736+0000 7ff49259c640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff47406c9b0 0x7ff47406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.736+0000 7ff49259c640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff47406c9b0 0x7ff47406ee70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff4940b14f0 tx=0x7ff484008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.740+0000 7ff48affd640 1 -- 172.21.15.12:0/2030745852 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff48c07bc80 con 0x7ff4940b0710 2024-04-15T15:22:57.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.924+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4940b4e20 con 0x7ff4940b0710 2024-04-15T15:22:57.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.925+0000 7ff48affd640 1 -- 172.21.15.12:0/2030745852 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16955 (secure 0 0 0) 0x7ff48c07ff80 con 0x7ff4940b0710 2024-04-15T15:22:57.928 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:57.928 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":41,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:22:51.946994+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:57.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff47406c9b0 msgr2=0x7ff47406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:57.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff47406c9b0 0x7ff47406ee70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff4940b14f0 tx=0x7ff484008040 comp rx=0 tx=0).stop 2024-04-15T15:22:57.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4940b0710 msgr2=0x7ff4940b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:57.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4940b0710 0x7ff4940b0bc0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00a8a0 tx=0x7ff48c001dc0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 shutdown_connections 2024-04-15T15:22:57.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4940b0710 0x7ff4940b0bc0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7ff47406c9b0 0x7ff47406ee70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4940caa00 0x7ff4940b79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 --2- 172.21.15.12:0/2030745852 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7ff480004d40 0x7ff4941f8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 >> 172.21.15.12:0/2030745852 conn(0x7ff4940d2460 msgr2=0x7ff4940b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 shutdown_connections 2024-04-15T15:22:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:57.927+0000 7ff498a9a640 1 -- 172.21.15.12:0/2030745852 wait complete. 2024-04-15T15:22:57.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:22:57.941 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:58.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.046+0000 7f922eb7c640 1 Processor -- start 2024-04-15T15:22:58.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.046+0000 7f922eb7c640 1 -- start start 2024-04-15T15:22:58.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922eb7c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 0x7f9228160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922eb7c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f92280701f0 0x7f922815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922eb7c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f922809adb0 0x7f92280d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922eb7c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f922809ba90 con 0x7f92280701f0 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922eb7c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9228160070 con 0x7f922809adb0 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922eb7c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f92280d08c0 con 0x7f922809a9d0 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f9227fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f92280701f0 0x7f922815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f9227fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f92280701f0 0x7f922815fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41770/0 (socket says 172.21.15.12:41770) 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f9227fff640 1 -- 172.21.15.12:0/687664069 learned_addr learned my addr 172.21.15.12:0/687664069 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922c918640 1 --2- 172.21.15.12:0/687664069 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 0x7f9228160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922d119640 1 --2- 172.21.15.12:0/687664069 >> v2:172.21.15.55:3300/0 conn(0x7f922809adb0 0x7f92280d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f9227fff640 1 -- 172.21.15.12:0/687664069 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 msgr2=0x7f9228160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f9227fff640 1 --2- 172.21.15.12:0/687664069 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 0x7f9228160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f9227fff640 1 -- 172.21.15.12:0/687664069 >> v2:172.21.15.55:3300/0 conn(0x7f922809adb0 msgr2=0x7f92280d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:58.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f9227fff640 1 --2- 172.21.15.12:0/687664069 >> v2:172.21.15.55:3300/0 conn(0x7f922809adb0 0x7f92280d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f9227fff640 1 -- 172.21.15.12:0/687664069 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9228070720 con 0x7f92280701f0 2024-04-15T15:22:58.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.047+0000 7f922d119640 1 --2- 172.21.15.12:0/687664069 >> v2:172.21.15.55:3300/0 conn(0x7f922809adb0 0x7f92280d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:58.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.048+0000 7f9227fff640 1 --2- 172.21.15.12:0/687664069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92280701f0 0x7f922815fb30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9214009870 tx=0x7f921402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49f049ad556107 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.048+0000 7f92277fe640 1 -- 172.21.15.12:0/687664069 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214035020 con 0x7f92280701f0 2024-04-15T15:22:58.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.048+0000 7f92277fe640 1 -- 172.21.15.12:0/687664069 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9214031640 con 0x7f92280701f0 2024-04-15T15:22:58.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.048+0000 7f92277fe640 1 -- 172.21.15.12:0/687664069 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921402d040 con 0x7f92280701f0 2024-04-15T15:22:58.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 -- 172.21.15.12:0/687664069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92280701f0 msgr2=0x7f922815fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 --2- 172.21.15.12:0/687664069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92280701f0 0x7f922815fb30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9214009870 tx=0x7f921402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 -- 172.21.15.12:0/687664069 shutdown_connections 2024-04-15T15:22:58.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 --2- 172.21.15.12:0/687664069 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92280701f0 0x7f922815fb30 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 --2- 172.21.15.12:0/687664069 >> v2:172.21.15.55:3300/0 conn(0x7f922809adb0 0x7f92280d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 --2- 172.21.15.12:0/687664069 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 0x7f9228160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 -- 172.21.15.12:0/687664069 >> 172.21.15.12:0/687664069 conn(0x7f92280d2460 msgr2=0x7f92280d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:58.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 -- 172.21.15.12:0/687664069 shutdown_connections 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.049+0000 7f922eb7c640 1 -- 172.21.15.12:0/687664069 wait complete. 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.050+0000 7f922eb7c640 1 Processor -- start 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.050+0000 7f922eb7c640 1 -- start start 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922eb7c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92280701f0 0x7f922815cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922eb7c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 0x7f92280b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922eb7c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f922809adb0 0x7f92280b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922eb7c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9228071050 con 0x7f922809adb0 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922eb7c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9228070d70 con 0x7f92280701f0 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922eb7c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f92280b1c60 con 0x7f922809a9d0 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f922809adb0 0x7f92280b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f922809adb0 0x7f92280b72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41782/0 (socket says 172.21.15.12:41782) 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 -- 172.21.15.12:0/1852503875 learned_addr learned my addr 172.21.15.12:0/1852503875 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922c918640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92280701f0 0x7f922815cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f9227fff640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 0x7f92280b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 -- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 msgr2=0x7f92280b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 0x7f92280b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 -- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92280701f0 msgr2=0x7f922815cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92280701f0 0x7f922815cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 -- 172.21.15.12:0/1852503875 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922815d470 con 0x7f922809adb0 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.051+0000 7f922d119640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f922809adb0 0x7f92280b72c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9220007e20 tx=0x7f922000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.052+0000 7f92257fa640 1 -- 172.21.15.12:0/1852503875 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922001a070 con 0x7f922809adb0 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.052+0000 7f92257fa640 1 -- 172.21.15.12:0/1852503875 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9220003b90 con 0x7f922809adb0 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.052+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f922815d750 con 0x7f922809adb0 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.052+0000 7f92257fa640 1 -- 172.21.15.12:0/1852503875 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922000de80 con 0x7f922809adb0 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.052+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92280b5ce0 con 0x7f922809adb0 2024-04-15T15:22:58.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.053+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92280c8c50 con 0x7f922809adb0 2024-04-15T15:22:58.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.053+0000 7f92257fa640 1 -- 172.21.15.12:0/1852503875 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f922001d070 con 0x7f922809adb0 2024-04-15T15:22:58.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.053+0000 7f92257fa640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f920c06c9b0 0x7f920c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.054+0000 7f92257fa640 1 -- 172.21.15.12:0/1852503875 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f922008bf70 con 0x7f922809adb0 2024-04-15T15:22:58.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.054+0000 7f922c918640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f920c06c9b0 0x7f920c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.054+0000 7f922c918640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f920c06c9b0 0x7f920c06ee70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f921c009ac0 tx=0x7f921c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.058+0000 7f92257fa640 1 -- 172.21.15.12:0/1852503875 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9220010070 con 0x7f922809adb0 2024-04-15T15:22:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.241+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f92280689e0 con 0x7f922809adb0 2024-04-15T15:22:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.242+0000 7f92257fa640 1 -- 172.21.15.12:0/1852503875 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f9220059b30 con 0x7f922809adb0 2024-04-15T15:22:58.244 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.243+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f920c06c9b0 msgr2=0x7f920c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.243+0000 7f922eb7c640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f920c06c9b0 0x7f920c06ee70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f921c009ac0 tx=0x7f921c009190 comp rx=0 tx=0).stop 2024-04-15T15:22:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.243+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f922809adb0 msgr2=0x7f92280b72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.243+0000 7f922eb7c640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f922809adb0 0x7f92280b72c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9220007e20 tx=0x7f922000d4a0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.244+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 shutdown_connections 2024-04-15T15:22:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.244+0000 7f922eb7c640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f922809adb0 0x7f92280b72c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.244+0000 7f922eb7c640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f922809a9d0 0x7f92280b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.244+0000 7f922eb7c640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f920c06c9b0 0x7f920c06ee70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.244+0000 7f922eb7c640 1 --2- 172.21.15.12:0/1852503875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92280701f0 0x7f922815cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.244+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 >> 172.21.15.12:0/1852503875 conn(0x7f92280d2460 msgr2=0x7f922815bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.244+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 shutdown_connections 2024-04-15T15:22:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.244+0000 7f922eb7c640 1 -- 172.21.15.12:0/1852503875 wait complete. 2024-04-15T15:22:58.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:58.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.363+0000 7f2f93e88640 1 Processor -- start 2024-04-15T15:22:58.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.363+0000 7f2f93e88640 1 -- start start 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f93e88640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2f8c15ecb0 0x7f2f8c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f93e88640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f8c0b1d30 0x7f2f8c0af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f93e88640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f8c0b10d0 con 0x7f2f8c0b4350 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f93e88640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2f8c15f090 con 0x7f2f8c15ecb0 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f93e88640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2f8c0cdf80 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f91c24640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2f8c15ecb0 0x7f2f8c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f91c24640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2f8c15ecb0 0x7f2f8c0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41798/0 (socket says 172.21.15.12:41798) 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f91c24640 1 -- 172.21.15.12:0/2225089075 learned_addr learned my addr 172.21.15.12:0/2225089075 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f91423640 1 --1- 172.21.15.12:0/2225089075 >> v1:172.21.15.12:6789/0 conn(0x7f2f8c0b4350 0x7f2f8c0b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47396/0 (socket says 172.21.15.12:47396) 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f92425640 1 --2- 172.21.15.12:0/2225089075 >> v2:172.21.15.55:3300/0 conn(0x7f2f8c0b1d30 0x7f2f8c0af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f91c24640 1 -- 172.21.15.12:0/2225089075 >> v2:172.21.15.55:3300/0 conn(0x7f2f8c0b1d30 msgr2=0x7f2f8c0af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:58.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f91c24640 1 --2- 172.21.15.12:0/2225089075 >> v2:172.21.15.55:3300/0 conn(0x7f2f8c0b1d30 0x7f2f8c0af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f91c24640 1 -- 172.21.15.12:0/2225089075 >> v1:172.21.15.12:6789/0 conn(0x7f2f8c0b4350 legacy=0x7f2f8c0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.364+0000 7f2f91c24640 1 -- 172.21.15.12:0/2225089075 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8c0af920 con 0x7f2f8c15ecb0 2024-04-15T15:22:58.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.365+0000 7f2f91c24640 1 --2- 172.21.15.12:0/2225089075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c15ecb0 0x7f2f8c0b0b90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2f88009870 tx=0x7f2f8802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86732c39cd287f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.365+0000 7f2f90c22640 1 -- 172.21.15.12:0/2225089075 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8802f840 con 0x7f2f8c15ecb0 2024-04-15T15:22:58.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.365+0000 7f2f90c22640 1 -- 172.21.15.12:0/2225089075 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8802f9e0 con 0x7f2f8c15ecb0 2024-04-15T15:22:58.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.365+0000 7f2f90c22640 1 -- 172.21.15.12:0/2225089075 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8802fdb0 con 0x7f2f8c15ecb0 2024-04-15T15:22:58.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.366+0000 7f2f93e88640 1 -- 172.21.15.12:0/2225089075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c15ecb0 msgr2=0x7f2f8c0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.366+0000 7f2f93e88640 1 --2- 172.21.15.12:0/2225089075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c15ecb0 0x7f2f8c0b0b90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2f88009870 tx=0x7f2f8802ba20 comp rx=0 tx=0).stop 2024-04-15T15:22:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.366+0000 7f2f93e88640 1 -- 172.21.15.12:0/2225089075 shutdown_connections 2024-04-15T15:22:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.366+0000 7f2f93e88640 1 --2- 172.21.15.12:0/2225089075 >> v2:172.21.15.55:3300/0 conn(0x7f2f8c0b1d30 0x7f2f8c0af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.366+0000 7f2f93e88640 1 --2- 172.21.15.12:0/2225089075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c15ecb0 0x7f2f8c0b0b90 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.366+0000 7f2f93e88640 1 -- 172.21.15.12:0/2225089075 >> 172.21.15.12:0/2225089075 conn(0x7f2f8c0b6870 msgr2=0x7f2f8c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.366+0000 7f2f93e88640 1 -- 172.21.15.12:0/2225089075 shutdown_connections 2024-04-15T15:22:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.366+0000 7f2f93e88640 1 -- 172.21.15.12:0/2225089075 wait complete. 2024-04-15T15:22:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.367+0000 7f2f93e88640 1 Processor -- start 2024-04-15T15:22:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.367+0000 7f2f93e88640 1 -- start start 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f93e88640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c0b1d30 0x7f2f8c15c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f93e88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0b4350 0x7f2f8c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f93e88640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f8c15ecb0 0x7f2f8c0ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f93e88640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c0b47c0 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f93e88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c0afee0 con 0x7f2f8c0b4350 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f93e88640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2f8c161ee0 con 0x7f2f8c15ecb0 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c0b1d30 0x7f2f8c15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c0b1d30 0x7f2f8c15c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41808/0 (socket says 172.21.15.12:41808) 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 -- 172.21.15.12:0/3045592522 learned_addr learned my addr 172.21.15.12:0/3045592522 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91423640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0b4350 0x7f2f8c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 -- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f8c15ecb0 msgr2=0x7f2f8c0ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f92425640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f8c15ecb0 0x7f2f8c0ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f8c15ecb0 0x7f2f8c0ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 -- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0b4350 msgr2=0x7f2f8c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0b4350 0x7f2f8c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 -- 172.21.15.12:0/3045592522 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8c0ca870 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91423640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0b4350 0x7f2f8c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f92425640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f8c15ecb0 0x7f2f8c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f91c24640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c0b1d30 0x7f2f8c15c270 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2f88000c00 tx=0x7f2f88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f827fc640 1 -- 172.21.15.12:0/3045592522 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88037c90 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f827fc640 1 -- 172.21.15.12:0/3045592522 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f88046070 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f827fc640 1 -- 172.21.15.12:0/3045592522 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88041640 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f8c0cab50 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.368+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f8c1e8170 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.369+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f8c0689e0 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.370+0000 7f2f827fc640 1 -- 172.21.15.12:0/3045592522 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2f88040070 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.370+0000 7f2f827fc640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f7006c980 0x7f2f7006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.370+0000 7f2f827fc640 1 -- 172.21.15.12:0/3045592522 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f2f880ae560 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.371+0000 7f2f91423640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f7006c980 0x7f2f7006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.371+0000 7f2f91423640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f7006c980 0x7f2f7006ee40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c1eb400 tx=0x7f2f7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.375+0000 7f2f827fc640 1 -- 172.21.15.12:0/3045592522 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f88033030 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.559+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2f8c069210 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.559+0000 7f2f827fc640 1 -- 172.21.15.12:0/3045592522 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7f2f8807c6e0 con 0x7f2f8c0b1d30 2024-04-15T15:22:58.562 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:22:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f7006c980 msgr2=0x7f2f7006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f7006c980 0x7f2f7006ee40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c1eb400 tx=0x7f2f7c009040 comp rx=0 tx=0).stop 2024-04-15T15:22:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c0b1d30 msgr2=0x7f2f8c15c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c0b1d30 0x7f2f8c15c270 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2f88000c00 tx=0x7f2f88004730 comp rx=0 tx=0).stop 2024-04-15T15:22:58.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 shutdown_connections 2024-04-15T15:22:58.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2f8c15ecb0 0x7f2f8c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2f7006c980 0x7f2f7006ee40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0b4350 0x7f2f8c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 --2- 172.21.15.12:0/3045592522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c0b1d30 0x7f2f8c15c270 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 >> 172.21.15.12:0/3045592522 conn(0x7f2f8c0b6870 msgr2=0x7f2f8c0af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:58.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 shutdown_connections 2024-04-15T15:22:58.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.561+0000 7f2f93e88640 1 -- 172.21.15.12:0/3045592522 wait complete. 2024-04-15T15:22:58.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:22:58.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.679+0000 7fb3d7b00640 1 Processor -- start 2024-04-15T15:22:58.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.679+0000 7fb3d7b00640 1 -- start start 2024-04-15T15:22:58.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.680+0000 7fb3d7b00640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb3d0072d80 0x7fb3d0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.680+0000 7fb3d7b00640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3d015db40 con 0x7fb3d00af3e0 2024-04-15T15:22:58.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.680+0000 7fb3d7b00640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3d015cfc0 con 0x7fb3d00d0da0 2024-04-15T15:22:58.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.680+0000 7fb3d7b00640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d015d2a0 con 0x7fb3d0072d80 2024-04-15T15:22:58.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.680+0000 7fb3d589c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb3d0072d80 0x7fb3d0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.680+0000 7fb3d589c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb3d0072d80 0x7fb3d0071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41822/0 (socket says 172.21.15.12:41822) 2024-04-15T15:22:58.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.680+0000 7fb3d589c640 1 -- 172.21.15.12:0/2460083796 learned_addr learned my addr 172.21.15.12:0/2460083796 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:58.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.681+0000 7fb3d589c640 1 -- 172.21.15.12:0/2460083796 >> v1:172.21.15.55:6789/0 conn(0x7fb3d00d0da0 legacy=0x7fb3d006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:58.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.681+0000 7fb3d589c640 1 -- 172.21.15.12:0/2460083796 >> v1:172.21.15.12:6789/0 conn(0x7fb3d00af3e0 legacy=0x7fb3d00d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.681+0000 7fb3d589c640 1 -- 172.21.15.12:0/2460083796 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d015d580 con 0x7fb3d0072d80 2024-04-15T15:22:58.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.681+0000 7fb3d589c640 1 --2- 172.21.15.12:0/2460083796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d0072d80 0x7fb3d0071d10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009e50 tx=0x7fb3c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36096f29f29d5ade server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.681+0000 7fb3d489a640 1 -- 172.21.15.12:0/2460083796 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c002ca80 con 0x7fb3d0072d80 2024-04-15T15:22:58.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.681+0000 7fb3d489a640 1 -- 172.21.15.12:0/2460083796 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c002cc20 con 0x7fb3d0072d80 2024-04-15T15:22:58.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.682+0000 7fb3d489a640 1 -- 172.21.15.12:0/2460083796 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c00343f0 con 0x7fb3d0072d80 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.682+0000 7fb3d7b00640 1 -- 172.21.15.12:0/2460083796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d0072d80 msgr2=0x7fb3d0071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.682+0000 7fb3d7b00640 1 --2- 172.21.15.12:0/2460083796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d0072d80 0x7fb3d0071d10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009e50 tx=0x7fb3c002bee0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.682+0000 7fb3d7b00640 1 -- 172.21.15.12:0/2460083796 shutdown_connections 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.682+0000 7fb3d7b00640 1 --2- 172.21.15.12:0/2460083796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d0072d80 0x7fb3d0071d10 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.682+0000 7fb3d7b00640 1 -- 172.21.15.12:0/2460083796 >> 172.21.15.12:0/2460083796 conn(0x7fb3d006be00 msgr2=0x7fb3d006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.682+0000 7fb3d7b00640 1 -- 172.21.15.12:0/2460083796 shutdown_connections 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.682+0000 7fb3d7b00640 1 -- 172.21.15.12:0/2460083796 wait complete. 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.683+0000 7fb3d7b00640 1 Processor -- start 2024-04-15T15:22:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.683+0000 7fb3d7b00640 1 -- start start 2024-04-15T15:22:58.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d7b00640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3d0072d80 0x7fb3d00cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d7b00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d00af3e0 0x7fb3d015b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d7b00640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3d00d0da0 0x7fb3d015b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d7b00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d015cd10 con 0x7fb3d00af3e0 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d7b00640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d015cb90 con 0x7fb3d0072d80 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d7b00640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d00af840 con 0x7fb3d00d0da0 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d509b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d00af3e0 0x7fb3d015b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d509b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d00af3e0 0x7fb3d015b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41836/0 (socket says 172.21.15.12:41836) 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d509b640 1 -- 172.21.15.12:0/1008651820 learned_addr learned my addr 172.21.15.12:0/1008651820 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d589c640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3d0072d80 0x7fb3d00cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d609d640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3d00d0da0 0x7fb3d015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d509b640 1 -- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3d00d0da0 msgr2=0x7fb3d015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d509b640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3d00d0da0 0x7fb3d015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d509b640 1 -- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3d0072d80 msgr2=0x7fb3d00cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d509b640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3d0072d80 0x7fb3d00cf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d509b640 1 -- 172.21.15.12:0/1008651820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d00b5140 con 0x7fb3d00af3e0 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d589c640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3d0072d80 0x7fb3d00cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.684+0000 7fb3d609d640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3d00d0da0 0x7fb3d015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.685+0000 7fb3d509b640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d00af3e0 0x7fb3d015b150 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c400 tx=0x7fb3c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.685+0000 7fb3ce7fc640 1 -- 172.21.15.12:0/1008651820 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40052d0 con 0x7fb3d00af3e0 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.685+0000 7fb3ce7fc640 1 -- 172.21.15.12:0/1008651820 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c4005490 con 0x7fb3d00af3e0 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.685+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d00b3f50 con 0x7fb3d00af3e0 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.685+0000 7fb3ce7fc640 1 -- 172.21.15.12:0/1008651820 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4007260 con 0x7fb3d00af3e0 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.685+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d00b3c40 con 0x7fb3d00af3e0 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.686+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d00689e0 con 0x7fb3d00af3e0 2024-04-15T15:22:58.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.686+0000 7fb3ce7fc640 1 -- 172.21.15.12:0/1008651820 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb3c4007400 con 0x7fb3d00af3e0 2024-04-15T15:22:58.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.687+0000 7fb3ce7fc640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3bc06c980 0x7fb3bc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.687+0000 7fb3ce7fc640 1 -- 172.21.15.12:0/1008651820 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fb3c408b2c0 con 0x7fb3d00af3e0 2024-04-15T15:22:58.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.687+0000 7fb3d589c640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3bc06c980 0x7fb3bc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:58.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.688+0000 7fb3d589c640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3bc06c980 0x7fb3bc06ee40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0035000 tx=0x7fb3c0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:58.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.692+0000 7fb3ce7fc640 1 -- 172.21.15.12:0/1008651820 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3c4059550 con 0x7fb3d00af3e0 2024-04-15T15:22:58.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.875+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb3d00afbb0 con 0x7fb3d00af3e0 2024-04-15T15:22:58.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.876+0000 7fb3ce7fc640 1 -- 172.21.15.12:0/1008651820 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7fb3c4058e50 con 0x7fb3d00af3e0 2024-04-15T15:22:58.878 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3bc06c980 msgr2=0x7fb3bc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3bc06c980 0x7fb3bc06ee40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0035000 tx=0x7fb3c0030000 comp rx=0 tx=0).stop 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d00af3e0 msgr2=0x7fb3d015b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d00af3e0 0x7fb3d015b150 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c400 tx=0x7fb3c400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 shutdown_connections 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb3d00d0da0 0x7fb3d015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d00af3e0 0x7fb3d015b150 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb3bc06c980 0x7fb3bc06ee40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 --2- 172.21.15.12:0/1008651820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3d0072d80 0x7fb3d00cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.877+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 >> 172.21.15.12:0/1008651820 conn(0x7fb3d006be00 msgr2=0x7fb3d0074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.878+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 shutdown_connections 2024-04-15T15:22:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.878+0000 7fb3d7b00640 1 -- 172.21.15.12:0/1008651820 wait complete. 2024-04-15T15:22:58.891 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:22:58.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f656198f640 1 Processor -- start 2024-04-15T15:22:58.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f656198f640 1 -- start start 2024-04-15T15:22:58.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f656198f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c0701f0 0x7f655c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f656198f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f655c15fb30 0x7f655c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:58.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f656198f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c070720 con 0x7f655c09a9d0 2024-04-15T15:22:58.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f656198f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f655c09ba90 con 0x7f655c15fb30 2024-04-15T15:22:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f656198f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f655c0d08c0 con 0x7f655c0701f0 2024-04-15T15:22:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f655b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f655c15fb30 0x7f655c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.996+0000 7f655b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f655c15fb30 0x7f655c0d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41844/0 (socket says 172.21.15.12:41844) 2024-04-15T15:22:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.997+0000 7f655b7fe640 1 -- 172.21.15.12:0/2762144146 learned_addr learned my addr 172.21.15.12:0/2762144146 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.997+0000 7f655b7fe640 1 -- 172.21.15.12:0/2762144146 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c0701f0 msgr2=0x7f655c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.997+0000 7f655b7fe640 1 --2- 172.21.15.12:0/2762144146 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c0701f0 0x7f655c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.997+0000 7f655b7fe640 1 -- 172.21.15.12:0/2762144146 >> v1:172.21.15.55:6789/0 conn(0x7f655c09a9d0 legacy=0x7f655c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.997+0000 7f655b7fe640 1 -- 172.21.15.12:0/2762144146 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c070a90 con 0x7f655c15fb30 2024-04-15T15:22:59.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.997+0000 7f655b7fe640 1 --2- 172.21.15.12:0/2762144146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c15fb30 0x7f655c0d0d50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6554009e80 tx=0x7f655402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b319194d1d367c28 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f6559ffb640 1 -- 172.21.15.12:0/2762144146 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655402ca50 con 0x7f655c15fb30 2024-04-15T15:22:59.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f6559ffb640 1 -- 172.21.15.12:0/2762144146 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655402cbf0 con 0x7f655c15fb30 2024-04-15T15:22:59.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f6559ffb640 1 -- 172.21.15.12:0/2762144146 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65540343f0 con 0x7f655c15fb30 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f656198f640 1 -- 172.21.15.12:0/2762144146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c15fb30 msgr2=0x7f655c0d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f656198f640 1 --2- 172.21.15.12:0/2762144146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c15fb30 0x7f655c0d0d50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6554009e80 tx=0x7f655402beb0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f656198f640 1 -- 172.21.15.12:0/2762144146 shutdown_connections 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f656198f640 1 --2- 172.21.15.12:0/2762144146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c15fb30 0x7f655c0d0d50 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f656198f640 1 --2- 172.21.15.12:0/2762144146 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c0701f0 0x7f655c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f656198f640 1 -- 172.21.15.12:0/2762144146 >> 172.21.15.12:0/2762144146 conn(0x7f655c0d2460 msgr2=0x7f655c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f656198f640 1 -- 172.21.15.12:0/2762144146 shutdown_connections 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.998+0000 7f656198f640 1 -- 172.21.15.12:0/2762144146 wait complete. 2024-04-15T15:22:59.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:58.999+0000 7f656198f640 1 Processor -- start 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f656198f640 1 -- start start 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f656198f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f655c0701f0 0x7f655c15ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f656198f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c09a9d0 0x7f655c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f656198f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c15fb30 0x7f655c0d1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f656198f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f655c15ffc0 con 0x7f655c09a9d0 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f656198f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f655c071050 con 0x7f655c0701f0 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f656198f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f655c0b1c60 con 0x7f655c15fb30 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f655a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c09a9d0 0x7f655c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f655a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c09a9d0 0x7f655c0d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41852/0 (socket says 172.21.15.12:41852) 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.000+0000 7f655a7fc640 1 -- 172.21.15.12:0/2406553783 learned_addr learned my addr 172.21.15.12:0/2406553783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655b7fe640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c15fb30 0x7f655c0d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655affd640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f655c0701f0 0x7f655c15ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655a7fc640 1 -- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c15fb30 msgr2=0x7f655c0d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655a7fc640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c15fb30 0x7f655c0d1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655a7fc640 1 -- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f655c0701f0 msgr2=0x7f655c15ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655a7fc640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f655c0701f0 0x7f655c15ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655a7fc640 1 -- 172.21.15.12:0/2406553783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c1f56d0 con 0x7f655c09a9d0 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655affd640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f655c0701f0 0x7f655c15ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655b7fe640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c15fb30 0x7f655c0d1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f655a7fc640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c09a9d0 0x7f655c0d0d50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f654400b410 tx=0x7f654400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f6553fff640 1 -- 172.21.15.12:0/2406553783 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544015020 con 0x7f655c09a9d0 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f6553fff640 1 -- 172.21.15.12:0/2406553783 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6544011850 con 0x7f655c09a9d0 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655c1f59b0 con 0x7f655c09a9d0 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f6553fff640 1 -- 172.21.15.12:0/2406553783 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544011c40 con 0x7f655c09a9d0 2024-04-15T15:22:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.001+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655c1f5f70 con 0x7f655c09a9d0 2024-04-15T15:22:59.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.002+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f655c0689e0 con 0x7f655c09a9d0 2024-04-15T15:22:59.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.003+0000 7f6553fff640 1 -- 172.21.15.12:0/2406553783 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6544018020 con 0x7f655c09a9d0 2024-04-15T15:22:59.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.003+0000 7f6553fff640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f653c06c9b0 0x7f653c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.003+0000 7f6553fff640 1 -- 172.21.15.12:0/2406553783 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f654408bc90 con 0x7f655c09a9d0 2024-04-15T15:22:59.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.004+0000 7f655affd640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f653c06c9b0 0x7f653c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.004+0000 7f655affd640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f653c06c9b0 0x7f653c06ee70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f654c006fd0 tx=0x7f654c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.008+0000 7f6553fff640 1 -- 172.21.15.12:0/2406553783 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f654401f020 con 0x7f655c09a9d0 2024-04-15T15:22:59.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.203+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f655c0c92c0 con 0x7f655c09a9d0 2024-04-15T15:22:59.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.204+0000 7f6553fff640 1 -- 172.21.15.12:0/2406553783 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) ==== 80+0+0 (secure 0 0 0) 0x7f65440172c0 con 0x7f655c09a9d0 2024-04-15T15:22:59.206 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:22:59.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f653c06c9b0 msgr2=0x7f653c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f653c06c9b0 0x7f653c06ee70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f654c006fd0 tx=0x7f654c008040 comp rx=0 tx=0).stop 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c09a9d0 msgr2=0x7f655c0d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c09a9d0 0x7f655c0d0d50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f654400b410 tx=0x7f654400b8d0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 shutdown_connections 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f655c15fb30 0x7f655c0d1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f655c09a9d0 0x7f655c0d0d50 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f653c06c9b0 0x7f653c06ee70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 --2- 172.21.15.12:0/2406553783 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f655c0701f0 0x7f655c15ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 >> 172.21.15.12:0/2406553783 conn(0x7f655c0d2460 msgr2=0x7f655c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 shutdown_connections 2024-04-15T15:22:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.206+0000 7f656198f640 1 -- 172.21.15.12:0/2406553783 wait complete. 2024-04-15T15:22:59.219 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:22:59.219 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:22:59.239 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:22:59.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.341+0000 7fa0868df640 1 Processor -- start 2024-04-15T15:22:59.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.341+0000 7fa0868df640 1 -- start start 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa0868df640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa080071d10 0x7fa0800d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa0868df640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0800d0bd0 0x7fa08006e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa0868df640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08015db40 con 0x7fa080072d80 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa0868df640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa0800d1040 con 0x7fa0800d0bd0 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa0868df640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa080072170 con 0x7fa080071d10 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa084e7c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0800d0bd0 0x7fa08006e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa084e7c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0800d0bd0 0x7fa08006e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41858/0 (socket says 172.21.15.12:41858) 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa084e7c640 1 -- 172.21.15.12:0/822012127 learned_addr learned my addr 172.21.15.12:0/822012127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa07f7fe640 1 --2- 172.21.15.12:0/822012127 >> v2:172.21.15.55:3300/0 conn(0x7fa080071d10 0x7fa0800d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa084e7c640 1 -- 172.21.15.12:0/822012127 >> v2:172.21.15.55:3300/0 conn(0x7fa080071d10 msgr2=0x7fa0800d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:59.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa084e7c640 1 --2- 172.21.15.12:0/822012127 >> v2:172.21.15.55:3300/0 conn(0x7fa080071d10 0x7fa0800d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa084e7c640 1 -- 172.21.15.12:0/822012127 >> v1:172.21.15.55:6789/0 conn(0x7fa080072d80 legacy=0x7fa0800af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.342+0000 7fa084e7c640 1 -- 172.21.15.12:0/822012127 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08015cfc0 con 0x7fa0800d0bd0 2024-04-15T15:22:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.343+0000 7fa084e7c640 1 --2- 172.21.15.12:0/822012127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0800d0bd0 0x7fa08006e060 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa0800d1240 tx=0x7fa07802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac8a6c4ab5ad753 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.343+0000 7fa07effd640 1 -- 172.21.15.12:0/822012127 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07802ca80 con 0x7fa0800d0bd0 2024-04-15T15:22:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.343+0000 7fa07effd640 1 -- 172.21.15.12:0/822012127 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07802cc20 con 0x7fa0800d0bd0 2024-04-15T15:22:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.343+0000 7fa07effd640 1 -- 172.21.15.12:0/822012127 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0780343f0 con 0x7fa0800d0bd0 2024-04-15T15:22:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.344+0000 7fa0868df640 1 -- 172.21.15.12:0/822012127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0800d0bd0 msgr2=0x7fa08006e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.344+0000 7fa0868df640 1 --2- 172.21.15.12:0/822012127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0800d0bd0 0x7fa08006e060 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa0800d1240 tx=0x7fa07802bee0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.344+0000 7fa0868df640 1 -- 172.21.15.12:0/822012127 shutdown_connections 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.344+0000 7fa0868df640 1 --2- 172.21.15.12:0/822012127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0800d0bd0 0x7fa08006e060 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.344+0000 7fa0868df640 1 --2- 172.21.15.12:0/822012127 >> v2:172.21.15.55:3300/0 conn(0x7fa080071d10 0x7fa0800d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.344+0000 7fa0868df640 1 -- 172.21.15.12:0/822012127 >> 172.21.15.12:0/822012127 conn(0x7fa08006be00 msgr2=0x7fa08006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.344+0000 7fa0868df640 1 -- 172.21.15.12:0/822012127 shutdown_connections 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.344+0000 7fa0868df640 1 -- 172.21.15.12:0/822012127 wait complete. 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.345+0000 7fa0868df640 1 Processor -- start 2024-04-15T15:22:59.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.345+0000 7fa0868df640 1 -- start start 2024-04-15T15:22:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa0868df640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa080071d10 0x7fa0800cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa0868df640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa080072d80 0x7fa0801f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa0868df640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0800d0bd0 0x7fa080155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa0868df640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa08015d860 con 0x7fa080071d10 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa0868df640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa08015d580 con 0x7fa080072d80 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa0868df640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa0800afaf0 con 0x7fa0800d0bd0 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa080071d10 0x7fa0800cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa080071d10 0x7fa0800cc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41870/0 (socket says 172.21.15.12:41870) 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07ffff640 1 -- 172.21.15.12:0/1415952036 learned_addr learned my addr 172.21.15.12:0/1415952036 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07f7fe640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa080072d80 0x7fa0801f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa084e7c640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0800d0bd0 0x7fa080155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07ffff640 1 -- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0800d0bd0 msgr2=0x7fa080155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07ffff640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0800d0bd0 0x7fa080155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07ffff640 1 -- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa080072d80 msgr2=0x7fa0801f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07ffff640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa080072d80 0x7fa0801f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.346+0000 7fa07ffff640 1 -- 172.21.15.12:0/1415952036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08009a5a0 con 0x7fa080071d10 2024-04-15T15:22:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.347+0000 7fa07ffff640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa080071d10 0x7fa0800cc670 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa07000c3c0 tx=0x7fa07000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.347+0000 7fa07cff9640 1 -- 172.21.15.12:0/1415952036 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0700052d0 con 0x7fa080071d10 2024-04-15T15:22:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.347+0000 7fa07cff9640 1 -- 172.21.15.12:0/1415952036 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa070005490 con 0x7fa080071d10 2024-04-15T15:22:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.347+0000 7fa07cff9640 1 -- 172.21.15.12:0/1415952036 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa070007260 con 0x7fa080071d10 2024-04-15T15:22:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.347+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08009b220 con 0x7fa080071d10 2024-04-15T15:22:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.347+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08009d8d0 con 0x7fa080071d10 2024-04-15T15:22:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.348+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08009d420 con 0x7fa080071d10 2024-04-15T15:22:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.348+0000 7fa07cff9640 1 -- 172.21.15.12:0/1415952036 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa070007400 con 0x7fa080071d10 2024-04-15T15:22:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.349+0000 7fa07cff9640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa06c06c980 0x7fa06c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.349+0000 7fa07cff9640 1 -- 172.21.15.12:0/1415952036 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fa07008b170 con 0x7fa080071d10 2024-04-15T15:22:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.349+0000 7fa07f7fe640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa06c06c980 0x7fa06c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.350+0000 7fa07f7fe640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa06c06c980 0x7fa06c06ee40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa0800cdaf0 tx=0x7fa074008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.353+0000 7fa07cff9640 1 -- 172.21.15.12:0/1415952036 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa070058db0 con 0x7fa080071d10 2024-04-15T15:22:59.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.539+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0800689e0 con 0x7fa080071d10 2024-04-15T15:22:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.540+0000 7fa07cff9640 1 -- 172.21.15.12:0/1415952036 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16955 (secure 0 0 0) 0x7fa07005d0b0 con 0x7fa080071d10 2024-04-15T15:22:59.543 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:22:59.543 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":41,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:22:51.946994+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa06c06c980 msgr2=0x7fa06c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa06c06c980 0x7fa06c06ee40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa0800cdaf0 tx=0x7fa074008040 comp rx=0 tx=0).stop 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa080071d10 msgr2=0x7fa0800cc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa080071d10 0x7fa0800cc670 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa07000c3c0 tx=0x7fa07000c880 comp rx=0 tx=0).stop 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 shutdown_connections 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0800d0bd0 0x7fa080155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa06c06c980 0x7fa06c06ee40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa080072d80 0x7fa0801f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 --2- 172.21.15.12:0/1415952036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa080071d10 0x7fa0800cc670 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 >> 172.21.15.12:0/1415952036 conn(0x7fa08006be00 msgr2=0x7fa0800b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 shutdown_connections 2024-04-15T15:22:59.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.542+0000 7fa0868df640 1 -- 172.21.15.12:0/1415952036 wait complete. 2024-04-15T15:22:59.556 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:22:59.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:22:59.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.661+0000 7f205f5ca640 1 Processor -- start 2024-04-15T15:22:59.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.661+0000 7f205f5ca640 1 -- start start 2024-04-15T15:22:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205f5ca640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f20580d0690 0x7f20580d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205f5ca640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f205815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205f5ca640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20580d0fb0 con 0x7f205815ef70 2024-04-15T15:22:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205f5ca640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f205815cb50 con 0x7f20580d0690 2024-04-15T15:22:59.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205f5ca640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f205815bfd0 con 0x7f20580b3160 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205db67640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f205815ef70 0x7f205815f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47414/0 (socket says 172.21.15.12:47414) 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205db67640 1 -- 172.21.15.12:0/1439439699 learned_addr learned my addr 172.21.15.12:0/1439439699 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205d366640 1 --2- 172.21.15.12:0/1439439699 >> v2:172.21.15.55:3300/0 conn(0x7f20580d0690 0x7f20580d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f205cb65640 1 --2- 172.21.15.12:0/1439439699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f205815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f2057fff640 1 -- 172.21.15.12:0/1439439699 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2814248400 0 0) 0x7f20580d0fb0 con 0x7f205815ef70 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.662+0000 7f2057fff640 1 -- 172.21.15.12:0/1439439699 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2040003520 con 0x7f205815ef70 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.663+0000 7f205cb65640 1 -- 172.21.15.12:0/1439439699 >> v2:172.21.15.55:3300/0 conn(0x7f20580d0690 msgr2=0x7f20580d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.663+0000 7f205cb65640 1 --2- 172.21.15.12:0/1439439699 >> v2:172.21.15.55:3300/0 conn(0x7f20580d0690 0x7f20580d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.663+0000 7f2057fff640 1 -- 172.21.15.12:0/1439439699 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2677786680 0 0) 0x7f2040003520 con 0x7f205815ef70 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.663+0000 7f205cb65640 1 -- 172.21.15.12:0/1439439699 >> v1:172.21.15.12:6789/0 conn(0x7f205815ef70 legacy=0x7f205815f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.663+0000 7f205cb65640 1 -- 172.21.15.12:0/1439439699 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205815c2b0 con 0x7f20580b3160 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205cb65640 1 --2- 172.21.15.12:0/1439439699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f205815b690 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f204c0097a0 tx=0x7f204c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe65dea3aa641e7e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f2057fff640 1 -- 172.21.15.12:0/1439439699 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c037070 con 0x7f20580b3160 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f2057fff640 1 -- 172.21.15.12:0/1439439699 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f204c030510 con 0x7f20580b3160 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f2057fff640 1 -- 172.21.15.12:0/1439439699 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20480019f0 con 0x7f20580b3160 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205f5ca640 1 -- 172.21.15.12:0/1439439699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 msgr2=0x7f205815b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205f5ca640 1 --2- 172.21.15.12:0/1439439699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f205815b690 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f204c0097a0 tx=0x7f204c02b790 comp rx=0 tx=0).stop 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205f5ca640 1 -- 172.21.15.12:0/1439439699 shutdown_connections 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205f5ca640 1 --2- 172.21.15.12:0/1439439699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f205815b690 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205f5ca640 1 --2- 172.21.15.12:0/1439439699 >> v2:172.21.15.55:3300/0 conn(0x7f20580d0690 0x7f20580d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205f5ca640 1 -- 172.21.15.12:0/1439439699 >> 172.21.15.12:0/1439439699 conn(0x7f205806be00 msgr2=0x7f205806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205f5ca640 1 -- 172.21.15.12:0/1439439699 shutdown_connections 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.664+0000 7f205f5ca640 1 -- 172.21.15.12:0/1439439699 wait complete. 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205f5ca640 1 Processor -- start 2024-04-15T15:22:59.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205f5ca640 1 -- start start 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205f5ca640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f20580cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205f5ca640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20580d0690 0x7f20580b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205f5ca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205815ef70 0x7f2058158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205f5ca640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f20580b4fb0 con 0x7f205815ef70 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205f5ca640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f205815c870 con 0x7f20580d0690 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205f5ca640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f205815bbd0 con 0x7f20580b3160 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205db67640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205815ef70 0x7f2058158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205db67640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205815ef70 0x7f2058158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41884/0 (socket says 172.21.15.12:41884) 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.666+0000 7f205db67640 1 -- 172.21.15.12:0/3848505290 learned_addr learned my addr 172.21.15.12:0/3848505290 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205d366640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f20580cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205db67640 1 -- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 msgr2=0x7f20580cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205db67640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f20580cec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205cb65640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20580d0690 0x7f20580b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205db67640 1 -- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20580d0690 msgr2=0x7f20580b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205db67640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20580d0690 0x7f20580b1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205db67640 1 -- 172.21.15.12:0/3848505290 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20580cf170 con 0x7f205815ef70 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205db67640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205815ef70 0x7f2058158a00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f204800cc90 tx=0x7f204800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f2055ffb640 1 -- 172.21.15.12:0/3848505290 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204800ce40 con 0x7f205815ef70 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f2055ffb640 1 -- 172.21.15.12:0/3848505290 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2048012400 con 0x7f205815ef70 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20580ce8a0 con 0x7f205815ef70 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f2055ffb640 1 -- 172.21.15.12:0/3848505290 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20480127f0 con 0x7f205815ef70 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.667+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20580cd3f0 con 0x7f205815ef70 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.668+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20580689e0 con 0x7f205815ef70 2024-04-15T15:22:59.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.669+0000 7f2055ffb640 1 -- 172.21.15.12:0/3848505290 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2048012990 con 0x7f205815ef70 2024-04-15T15:22:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.669+0000 7f2055ffb640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f204406c980 0x7f204406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.669+0000 7f2055ffb640 1 -- 172.21.15.12:0/3848505290 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f204808c540 con 0x7f205815ef70 2024-04-15T15:22:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.669+0000 7f205d366640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f204406c980 0x7f204406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.670+0000 7f205d366640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f204406c980 0x7f204406ee40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2050009970 tx=0x7f2050008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.673+0000 7f2055ffb640 1 -- 172.21.15.12:0/3848505290 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f204805a7e0 con 0x7f205815ef70 2024-04-15T15:22:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.857+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f20580739e0 con 0x7f205815ef70 2024-04-15T15:22:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.857+0000 7f2055ffb640 1 -- 172.21.15.12:0/3848505290 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f204805a0e0 con 0x7f205815ef70 2024-04-15T15:22:59.860 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:22:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f204406c980 msgr2=0x7f204406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f204406c980 0x7f204406ee40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2050009970 tx=0x7f2050008040 comp rx=0 tx=0).stop 2024-04-15T15:22:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205815ef70 msgr2=0x7f2058158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205815ef70 0x7f2058158a00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f204800cc90 tx=0x7f204800a430 comp rx=0 tx=0).stop 2024-04-15T15:22:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 shutdown_connections 2024-04-15T15:22:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f205815ef70 0x7f2058158a00 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f204406c980 0x7f204406ee40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20580d0690 0x7f20580b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 --2- 172.21.15.12:0/3848505290 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f20580b3160 0x7f20580cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 >> 172.21.15.12:0/3848505290 conn(0x7f205806be00 msgr2=0x7f205806dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 shutdown_connections 2024-04-15T15:22:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.860+0000 7f205f5ca640 1 -- 172.21.15.12:0/3848505290 wait complete. 2024-04-15T15:22:59.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:22:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.978+0000 7fa74fa35640 1 Processor -- start 2024-04-15T15:22:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.978+0000 7fa74fa35640 1 -- start start 2024-04-15T15:22:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74fa35640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7480ccff0 0x7fa7480cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74fa35640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 0x7fa7480c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74fa35640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7480cde30 con 0x7fa7480d04f0 2024-04-15T15:22:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74fa35640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa7480ce110 con 0x7fa7480ccff0 2024-04-15T15:22:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74fa35640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa7480c61b0 con 0x7fa7480cd910 2024-04-15T15:22:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74d7d1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7480ccff0 0x7fa7480cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74d7d1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa7480ccff0 0x7fa7480cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41898/0 (socket says 172.21.15.12:41898) 2024-04-15T15:22:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74d7d1640 1 -- 172.21.15.12:0/3651685783 learned_addr learned my addr 172.21.15.12:0/3651685783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.979+0000 7fa74cfd0640 1 --2- 172.21.15.12:0/3651685783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 0x7fa7480c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.980+0000 7fa74d7d1640 1 -- 172.21.15.12:0/3651685783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 msgr2=0x7fa7480c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:22:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.980+0000 7fa74d7d1640 1 --2- 172.21.15.12:0/3651685783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 0x7fa7480c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.980+0000 7fa74d7d1640 1 -- 172.21.15.12:0/3651685783 >> v1:172.21.15.12:6789/0 conn(0x7fa7480d04f0 legacy=0x7fa7480c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.980+0000 7fa74d7d1640 1 -- 172.21.15.12:0/3651685783 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7480c6490 con 0x7fa7480ccff0 2024-04-15T15:22:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.980+0000 7fa74d7d1640 1 --2- 172.21.15.12:0/3651685783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa7480cd3d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa738009e50 tx=0x7fa73802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4766d7fac440edbf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa747fff640 1 -- 172.21.15.12:0/3651685783 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73802cb00 con 0x7fa7480ccff0 2024-04-15T15:22:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa747fff640 1 -- 172.21.15.12:0/3651685783 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73802cca0 con 0x7fa7480ccff0 2024-04-15T15:22:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa747fff640 1 -- 172.21.15.12:0/3651685783 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738034610 con 0x7fa7480ccff0 2024-04-15T15:22:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa74fa35640 1 -- 172.21.15.12:0/3651685783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 msgr2=0x7fa7480cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa74fa35640 1 --2- 172.21.15.12:0/3651685783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa7480cd3d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa738009e50 tx=0x7fa73802bf60 comp rx=0 tx=0).stop 2024-04-15T15:22:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa74fa35640 1 -- 172.21.15.12:0/3651685783 shutdown_connections 2024-04-15T15:22:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa74fa35640 1 --2- 172.21.15.12:0/3651685783 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 0x7fa7480c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa74fa35640 1 --2- 172.21.15.12:0/3651685783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa7480cd3d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.983+0000 7fa74fa35640 1 -- 172.21.15.12:0/3651685783 >> 172.21.15.12:0/3651685783 conn(0x7fa7480b3f40 msgr2=0x7fa7480b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:22:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.984+0000 7fa74fa35640 1 -- 172.21.15.12:0/3651685783 shutdown_connections 2024-04-15T15:22:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.984+0000 7fa74fa35640 1 -- 172.21.15.12:0/3651685783 wait complete. 2024-04-15T15:22:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.985+0000 7fa74fa35640 1 Processor -- start 2024-04-15T15:22:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.985+0000 7fa74fa35640 1 -- start start 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.985+0000 7fa74fa35640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa748151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.985+0000 7fa74fa35640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 0x7fa74815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.985+0000 7fa74fa35640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7480d04f0 0x7fa7481588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.985+0000 7fa74fa35640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa74809bb60 con 0x7fa7480ccff0 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.985+0000 7fa74fa35640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa74809b9e0 con 0x7fa7480d04f0 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.985+0000 7fa74fa35640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa74809bce0 con 0x7fa7480cd910 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa748151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa748151490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41900/0 (socket says 172.21.15.12:41900) 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 -- 172.21.15.12:0/2378395149 learned_addr learned my addr 172.21.15.12:0/2378395149 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:22:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74cfd0640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 0x7fa74815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74dfd2640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7480d04f0 0x7fa7481588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 -- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 msgr2=0x7fa74815ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 0x7fa74815ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 -- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7480d04f0 msgr2=0x7fa7481588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7480d04f0 0x7fa7481588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 -- 172.21.15.12:0/2378395149 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7481555b0 con 0x7fa7480ccff0 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74d7d1640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa748151490 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa738000f80 tx=0x7fa738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa745ffb640 1 -- 172.21.15.12:0/2378395149 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738043020 con 0x7fa7480ccff0 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa745ffb640 1 -- 172.21.15.12:0/2378395149 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73802ce50 con 0x7fa7480ccff0 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa745ffb640 1 -- 172.21.15.12:0/2378395149 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738009510 con 0x7fa7480ccff0 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.986+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa748155890 con 0x7fa7480ccff0 2024-04-15T15:22:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.987+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7480b53d0 con 0x7fa7480ccff0 2024-04-15T15:22:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.987+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7480b51c0 con 0x7fa7480ccff0 2024-04-15T15:22:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.988+0000 7fa745ffb640 1 -- 172.21.15.12:0/2378395149 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa738046020 con 0x7fa7480ccff0 2024-04-15T15:22:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.988+0000 7fa745ffb640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa73406c980 0x7fa73406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:22:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.988+0000 7fa745ffb640 1 -- 172.21.15.12:0/2378395149 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fa7380ae370 con 0x7fa7480ccff0 2024-04-15T15:22:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.989+0000 7fa74cfd0640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa73406c980 0x7fa73406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:22:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.989+0000 7fa74cfd0640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa73406c980 0x7fa73406ee40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa7481f5be0 tx=0x7fa73c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:22:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:22:59.993+0000 7fa745ffb640 1 -- 172.21.15.12:0/2378395149 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa73804d030 con 0x7fa7480ccff0 2024-04-15T15:23:00.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.176+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa7480690f0 con 0x7fa7480ccff0 2024-04-15T15:23:00.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.176+0000 7fa745ffb640 1 -- 172.21.15.12:0/2378395149 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7fa73803c350 con 0x7fa7480ccff0 2024-04-15T15:23:00.178 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa73406c980 msgr2=0x7fa73406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa73406c980 0x7fa73406ee40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa7481f5be0 tx=0x7fa73c008040 comp rx=0 tx=0).stop 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 msgr2=0x7fa748151490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa748151490 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa738000f80 tx=0x7fa738004a40 comp rx=0 tx=0).stop 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 shutdown_connections 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa73406c980 0x7fa73406ee40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7480d04f0 0x7fa7481588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa7480cd910 0x7fa74815ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 --2- 172.21.15.12:0/2378395149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7480ccff0 0x7fa748151490 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 >> 172.21.15.12:0/2378395149 conn(0x7fa7480b3f40 msgr2=0x7fa7480d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 shutdown_connections 2024-04-15T15:23:00.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.178+0000 7fa74fa35640 1 -- 172.21.15.12:0/2378395149 wait complete. 2024-04-15T15:23:00.191 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:23:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.297+0000 7fa0c780d640 1 Processor -- start 2024-04-15T15:23:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.297+0000 7fa0c780d640 1 -- start start 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c780d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0c00d0690 0x7fa0c00d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c780d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0c00b3160 0x7fa0c015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c780d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0c00d0fb0 con 0x7fa0c015ef70 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c780d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa0c015cb50 con 0x7fa0c00d0690 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c780d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa0c015bfd0 con 0x7fa0c00b3160 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c55a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0c00d0690 0x7fa0c00d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c55a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0c00d0690 0x7fa0c00d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41904/0 (socket says 172.21.15.12:41904) 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c55a9640 1 -- 172.21.15.12:0/734176598 learned_addr learned my addr 172.21.15.12:0/734176598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c4da8640 1 --2- 172.21.15.12:0/734176598 >> v2:172.21.15.55:3300/0 conn(0x7fa0c00b3160 0x7fa0c015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c55a9640 1 -- 172.21.15.12:0/734176598 >> v2:172.21.15.55:3300/0 conn(0x7fa0c00b3160 msgr2=0x7fa0c015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c55a9640 1 --2- 172.21.15.12:0/734176598 >> v2:172.21.15.55:3300/0 conn(0x7fa0c00b3160 0x7fa0c015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c55a9640 1 -- 172.21.15.12:0/734176598 >> v1:172.21.15.55:6789/0 conn(0x7fa0c015ef70 legacy=0x7fa0c015f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.298+0000 7fa0c55a9640 1 -- 172.21.15.12:0/734176598 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c015c2b0 con 0x7fa0c00d0690 2024-04-15T15:23:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.299+0000 7fa0c55a9640 1 --2- 172.21.15.12:0/734176598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00d0690 0x7fa0c00d0a70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa0c006fec0 tx=0x7fa0bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de56c3a32b34ac51 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.299+0000 7fa0b7fff640 1 -- 172.21.15.12:0/734176598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc034070 con 0x7fa0c00d0690 2024-04-15T15:23:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.299+0000 7fa0b7fff640 1 -- 172.21.15.12:0/734176598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0bc030860 con 0x7fa0c00d0690 2024-04-15T15:23:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.299+0000 7fa0b7fff640 1 -- 172.21.15.12:0/734176598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc030c30 con 0x7fa0c00d0690 2024-04-15T15:23:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.300+0000 7fa0c780d640 1 -- 172.21.15.12:0/734176598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00d0690 msgr2=0x7fa0c00d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.300+0000 7fa0c780d640 1 --2- 172.21.15.12:0/734176598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00d0690 0x7fa0c00d0a70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa0c006fec0 tx=0x7fa0bc02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.300+0000 7fa0c780d640 1 -- 172.21.15.12:0/734176598 shutdown_connections 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.300+0000 7fa0c780d640 1 --2- 172.21.15.12:0/734176598 >> v2:172.21.15.55:3300/0 conn(0x7fa0c00b3160 0x7fa0c015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.300+0000 7fa0c780d640 1 --2- 172.21.15.12:0/734176598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00d0690 0x7fa0c00d0a70 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.300+0000 7fa0c780d640 1 -- 172.21.15.12:0/734176598 >> 172.21.15.12:0/734176598 conn(0x7fa0c006be00 msgr2=0x7fa0c006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.300+0000 7fa0c780d640 1 -- 172.21.15.12:0/734176598 shutdown_connections 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.300+0000 7fa0c780d640 1 -- 172.21.15.12:0/734176598 wait complete. 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.301+0000 7fa0c780d640 1 Processor -- start 2024-04-15T15:23:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.301+0000 7fa0c780d640 1 -- start start 2024-04-15T15:23:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c780d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00b3160 0x7fa0c0157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c780d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0c00d0690 0x7fa0c0155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c780d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0c015ef70 0x7fa0c01555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c780d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c00b4fb0 con 0x7fa0c00b3160 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c780d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c015c870 con 0x7fa0c00d0690 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c780d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c015bbd0 con 0x7fa0c015ef70 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00b3160 0x7fa0c0157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00b3160 0x7fa0c0157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41906/0 (socket says 172.21.15.12:41906) 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 -- 172.21.15.12:0/647184736 learned_addr learned my addr 172.21.15.12:0/647184736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c4da8640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0c00d0690 0x7fa0c0155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c5daa640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0c015ef70 0x7fa0c01555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 -- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0c015ef70 msgr2=0x7fa0c01555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0c015ef70 0x7fa0c01555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 -- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0c00d0690 msgr2=0x7fa0c0155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0c00d0690 0x7fa0c0155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 -- 172.21.15.12:0/647184736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c0155c30 con 0x7fa0c00b3160 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.302+0000 7fa0c55a9640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00b3160 0x7fa0c0157560 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc02b9e0 tx=0x7fa0bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.303+0000 7fa0b5ffb640 1 -- 172.21.15.12:0/647184736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc034040 con 0x7fa0c00b3160 2024-04-15T15:23:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.303+0000 7fa0b5ffb640 1 -- 172.21.15.12:0/647184736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0bc039d90 con 0x7fa0c00b3160 2024-04-15T15:23:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.303+0000 7fa0b5ffb640 1 -- 172.21.15.12:0/647184736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc043610 con 0x7fa0c00b3160 2024-04-15T15:23:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.303+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c0155f10 con 0x7fa0c00b3160 2024-04-15T15:23:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.303+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c01564d0 con 0x7fa0c00b3160 2024-04-15T15:23:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.304+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c00689e0 con 0x7fa0c00b3160 2024-04-15T15:23:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.304+0000 7fa0b5ffb640 1 -- 172.21.15.12:0/647184736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa0bc04a060 con 0x7fa0c00b3160 2024-04-15T15:23:00.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.305+0000 7fa0b5ffb640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0a406c9b0 0x7fa0a406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.305+0000 7fa0b5ffb640 1 -- 172.21.15.12:0/647184736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fa0bc0ae0e0 con 0x7fa0c00b3160 2024-04-15T15:23:00.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.305+0000 7fa0c4da8640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0a406c9b0 0x7fa0a406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.305+0000 7fa0c4da8640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0a406c9b0 0x7fa0a406ee70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0006fd0 tx=0x7fa0b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.309+0000 7fa0b5ffb640 1 -- 172.21.15.12:0/647184736 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0bc03e070 con 0x7fa0c00b3160 2024-04-15T15:23:00.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.494+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa0c00b1bd0 con 0x7fa0c00b3160 2024-04-15T15:23:00.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.495+0000 7fa0b5ffb640 1 -- 172.21.15.12:0/647184736 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7fa0bc037360 con 0x7fa0c00b3160 2024-04-15T15:23:00.497 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:23:00.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.496+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0a406c9b0 msgr2=0x7fa0a406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.496+0000 7fa0c780d640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0a406c9b0 0x7fa0a406ee70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0006fd0 tx=0x7fa0b0008040 comp rx=0 tx=0).stop 2024-04-15T15:23:00.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.496+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00b3160 msgr2=0x7fa0c0157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.496+0000 7fa0c780d640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00b3160 0x7fa0c0157560 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc02b9e0 tx=0x7fa0bc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.497+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 shutdown_connections 2024-04-15T15:23:00.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.497+0000 7fa0c780d640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa0c015ef70 0x7fa0c01555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.497+0000 7fa0c780d640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa0a406c9b0 0x7fa0a406ee70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.497+0000 7fa0c780d640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0c00d0690 0x7fa0c0155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.497+0000 7fa0c780d640 1 --2- 172.21.15.12:0/647184736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0c00b3160 0x7fa0c0157560 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.497+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 >> 172.21.15.12:0/647184736 conn(0x7fa0c006be00 msgr2=0x7fa0c00cee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:00.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.497+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 shutdown_connections 2024-04-15T15:23:00.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.497+0000 7fa0c780d640 1 -- 172.21.15.12:0/647184736 wait complete. 2024-04-15T15:23:00.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:23:00.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.613+0000 7f736e1bd640 1 Processor -- start 2024-04-15T15:23:00.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.613+0000 7f736e1bd640 1 -- start start 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f736e1bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73680701f0 0x7f7368160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f736e1bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f736815fb30 0x7f73680d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f736e1bd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7368070720 con 0x7f736809a9d0 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f736e1bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f736809ba90 con 0x7f736815fb30 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f736e1bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f73680d08c0 con 0x7f73680701f0 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f7367fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f736815fb30 0x7f73680d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f7367fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f736815fb30 0x7f73680d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41918/0 (socket says 172.21.15.12:41918) 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f7367fff640 1 -- 172.21.15.12:0/3722864077 learned_addr learned my addr 172.21.15.12:0/3722864077 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f7366ffd640 1 --2- 172.21.15.12:0/3722864077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73680701f0 0x7f7368160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f7367fff640 1 -- 172.21.15.12:0/3722864077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73680701f0 msgr2=0x7f7368160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:00.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f7367fff640 1 --2- 172.21.15.12:0/3722864077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73680701f0 0x7f7368160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f7367fff640 1 -- 172.21.15.12:0/3722864077 >> v1:172.21.15.55:6789/0 conn(0x7f736809a9d0 legacy=0x7f736809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.614+0000 7f7367fff640 1 -- 172.21.15.12:0/3722864077 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7368070a90 con 0x7f736815fb30 2024-04-15T15:23:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.615+0000 7f7367fff640 1 --2- 172.21.15.12:0/3722864077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736815fb30 0x7f73680d0d50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7360009e80 tx=0x7f736002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf15fe84fd9c72a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.615+0000 7f73667fc640 1 -- 172.21.15.12:0/3722864077 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736002caf0 con 0x7f736815fb30 2024-04-15T15:23:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.615+0000 7f73667fc640 1 -- 172.21.15.12:0/3722864077 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f736002cc90 con 0x7f736815fb30 2024-04-15T15:23:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.615+0000 7f73667fc640 1 -- 172.21.15.12:0/3722864077 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7360034610 con 0x7f736815fb30 2024-04-15T15:23:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.616+0000 7f736e1bd640 1 -- 172.21.15.12:0/3722864077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736815fb30 msgr2=0x7f73680d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.616+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3722864077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736815fb30 0x7f73680d0d50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7360009e80 tx=0x7f736002bf50 comp rx=0 tx=0).stop 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.616+0000 7f736e1bd640 1 -- 172.21.15.12:0/3722864077 shutdown_connections 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.616+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3722864077 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736815fb30 0x7f73680d0d50 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.616+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3722864077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f73680701f0 0x7f7368160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.616+0000 7f736e1bd640 1 -- 172.21.15.12:0/3722864077 >> 172.21.15.12:0/3722864077 conn(0x7f73680d2460 msgr2=0x7f73680d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.616+0000 7f736e1bd640 1 -- 172.21.15.12:0/3722864077 shutdown_connections 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.616+0000 7f736e1bd640 1 -- 172.21.15.12:0/3722864077 wait complete. 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.617+0000 7f736e1bd640 1 Processor -- start 2024-04-15T15:23:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.617+0000 7f736e1bd640 1 -- start start 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f736e1bd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73680701f0 0x7f7368154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f736e1bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736809a9d0 0x7f73680c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f736e1bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f736815fb30 0x7f73680b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f736e1bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f736815ffc0 con 0x7f736809a9d0 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f736e1bd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7368071050 con 0x7f73680701f0 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f736e1bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f73680b1c60 con 0x7f736815fb30 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7366ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736809a9d0 0x7f73680c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7366ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736809a9d0 0x7f73680c9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41920/0 (socket says 172.21.15.12:41920) 2024-04-15T15:23:00.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7366ffd640 1 -- 172.21.15.12:0/3377533670 learned_addr learned my addr 172.21.15.12:0/3377533670 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f73677fe640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73680701f0 0x7f7368154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7367fff640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f736815fb30 0x7f73680b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7366ffd640 1 -- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f736815fb30 msgr2=0x7f73680b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7366ffd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f736815fb30 0x7f73680b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7366ffd640 1 -- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73680701f0 msgr2=0x7f7368154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7366ffd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73680701f0 0x7f7368154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7366ffd640 1 -- 172.21.15.12:0/3377533670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73681f5520 con 0x7f736809a9d0 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f73677fe640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73680701f0 0x7f7368154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.618+0000 7f7367fff640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f736815fb30 0x7f73680b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.619+0000 7f7366ffd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736809a9d0 0x7f73680c9db0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f735c00d5c0 tx=0x7f735c00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.619+0000 7f734ffff640 1 -- 172.21.15.12:0/3377533670 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0125f0 con 0x7f736809a9d0 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.619+0000 7f734ffff640 1 -- 172.21.15.12:0/3377533670 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f735c019070 con 0x7f736809a9d0 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.619+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73681f5800 con 0x7f736809a9d0 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.619+0000 7f734ffff640 1 -- 172.21.15.12:0/3377533670 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0129d0 con 0x7f736809a9d0 2024-04-15T15:23:00.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.619+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73681f5dc0 con 0x7f736809a9d0 2024-04-15T15:23:00.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.620+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7368068250 con 0x7f736809a9d0 2024-04-15T15:23:00.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.620+0000 7f734ffff640 1 -- 172.21.15.12:0/3377533670 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f735c012b70 con 0x7f736809a9d0 2024-04-15T15:23:00.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.621+0000 7f734ffff640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f735406c980 0x7f735406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:00.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.621+0000 7f734ffff640 1 -- 172.21.15.12:0/3377533670 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f735c08c0b0 con 0x7f736809a9d0 2024-04-15T15:23:00.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.621+0000 7f73677fe640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f735406c980 0x7f735406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:00.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.622+0000 7f73677fe640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f735406c980 0x7f735406ee40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f73580015e0 tx=0x7f7358009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:00.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.625+0000 7f734ffff640 1 -- 172.21.15.12:0/3377533670 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7368068250 con 0x7f736809a9d0 2024-04-15T15:23:00.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.821+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f73680689e0 con 0x7f736809a9d0 2024-04-15T15:23:00.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.822+0000 7f734ffff640 1 -- 172.21.15.12:0/3377533670 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) ==== 80+0+0 (secure 0 0 0) 0x7f735c05a3d0 con 0x7f736809a9d0 2024-04-15T15:23:00.824 INFO:teuthology.orchestra.run.smithi012.stderr:listed 0 entries 2024-04-15T15:23:00.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f735406c980 msgr2=0x7f735406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f735406c980 0x7f735406ee40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f73580015e0 tx=0x7f7358009040 comp rx=0 tx=0).stop 2024-04-15T15:23:00.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736809a9d0 msgr2=0x7f73680c9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736809a9d0 0x7f73680c9db0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f735c00d5c0 tx=0x7f735c00da80 comp rx=0 tx=0).stop 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 shutdown_connections 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f736815fb30 0x7f73680b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f736809a9d0 0x7f73680c9db0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f735406c980 0x7f735406ee40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 --2- 172.21.15.12:0/3377533670 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73680701f0 0x7f7368154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 >> 172.21.15.12:0/3377533670 conn(0x7f73680d2460 msgr2=0x7f73680b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 shutdown_connections 2024-04-15T15:23:00.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:00.824+0000 7f736e1bd640 1 -- 172.21.15.12:0/3377533670 wait complete. 2024-04-15T15:23:00.837 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi055.front.sepia.ceph.com... 2024-04-15T15:23:00.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:00.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:23:00.869 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:ceph-fuse[45624]: fuse finished with error 0 and tester_r 0 2024-04-15T15:23:00.871 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:23:01.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:23:01.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:23:06.973 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:23:06.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:06.973 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:23:06.993 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:23:06.993 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:07.013 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20669 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:23:07.013 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-04-15T15:23:07.013 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:23:07.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:07.013 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:23:07.071 INFO:teuthology.orchestra.run.smithi012.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-15T15:23:07.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:23:07.072 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:23:07.072 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:07.091 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14740 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:23:07.091 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi055.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-04-15T15:23:07.091 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:23:07.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:07.092 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:23:07.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:23:07.150 INFO:teuthology.orchestra.run.smithi055.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-15T15:23:07.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:23:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.264+0000 7f41f3d3d640 1 Processor -- start 2024-04-15T15:23:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.264+0000 7f41f3d3d640 1 -- start start 2024-04-15T15:23:07.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f3d3d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec15e910 0x7f41ec0d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:07.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f3d3d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41ec0cfda0 0x7f41ec0d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:07.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f3d3d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec15ecf0 con 0x7f41ec0b6110 2024-04-15T15:23:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f3d3d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f41ec0d21d0 con 0x7f41ec0cfda0 2024-04-15T15:23:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f3d3d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f41ec0d24b0 con 0x7f41ec15e910 2024-04-15T15:23:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f12d8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41ec0cfda0 0x7f41ec0d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f12d8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41ec0cfda0 0x7f41ec0d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45644/0 (socket says 172.21.15.12:45644) 2024-04-15T15:23:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f12d8640 1 -- 172.21.15.12:0/4181125862 learned_addr learned my addr 172.21.15.12:0/4181125862 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f12d8640 1 -- 172.21.15.12:0/4181125862 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec15e910 msgr2=0x7f41ec0d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:07.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f1ad9640 1 --2- 172.21.15.12:0/4181125862 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec15e910 0x7f41ec0d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:07.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f12d8640 1 --2- 172.21.15.12:0/4181125862 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec15e910 0x7f41ec0d3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f12d8640 1 -- 172.21.15.12:0/4181125862 >> v1:172.21.15.12:6789/0 conn(0x7f41ec0b6110 legacy=0x7f41ec06cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:07.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.265+0000 7f41f12d8640 1 -- 172.21.15.12:0/4181125862 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec0d2790 con 0x7f41ec0cfda0 2024-04-15T15:23:07.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.266+0000 7f41f12d8640 1 --2- 172.21.15.12:0/4181125862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec0cfda0 0x7f41ec0d0200 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009870 tx=0x7f41dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79606bb96d00d4de server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:07.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.266+0000 7f41f0ad7640 1 -- 172.21.15.12:0/4181125862 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc034070 con 0x7f41ec0cfda0 2024-04-15T15:23:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.266+0000 7f41f0ad7640 1 -- 172.21.15.12:0/4181125862 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41dc030840 con 0x7f41ec0cfda0 2024-04-15T15:23:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.266+0000 7f41f0ad7640 1 -- 172.21.15.12:0/4181125862 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc030c10 con 0x7f41ec0cfda0 2024-04-15T15:23:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.267+0000 7f41f3d3d640 1 -- 172.21.15.12:0/4181125862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec0cfda0 msgr2=0x7f41ec0d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.267+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/4181125862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec0cfda0 0x7f41ec0d0200 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009870 tx=0x7f41dc02b980 comp rx=0 tx=0).stop 2024-04-15T15:23:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.267+0000 7f41f3d3d640 1 -- 172.21.15.12:0/4181125862 shutdown_connections 2024-04-15T15:23:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.267+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/4181125862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec0cfda0 0x7f41ec0d0200 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.267+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/4181125862 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec15e910 0x7f41ec0d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.267+0000 7f41f3d3d640 1 -- 172.21.15.12:0/4181125862 >> 172.21.15.12:0/4181125862 conn(0x7f41ec08bf60 msgr2=0x7f41ec0d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.267+0000 7f41f3d3d640 1 -- 172.21.15.12:0/4181125862 shutdown_connections 2024-04-15T15:23:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.267+0000 7f41f3d3d640 1 -- 172.21.15.12:0/4181125862 wait complete. 2024-04-15T15:23:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.268+0000 7f41f3d3d640 1 Processor -- start 2024-04-15T15:23:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.268+0000 7f41f3d3d640 1 -- start start 2024-04-15T15:23:07.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f3d3d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41ec0b6110 0x7f41ec158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:07.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f3d3d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec0cfda0 0x7f41ec15b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:07.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f3d3d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec15e910 0x7f41ec15c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:07.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f3d3d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec0d37b0 con 0x7f41ec15e910 2024-04-15T15:23:07.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f3d3d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec0d2d50 con 0x7f41ec0b6110 2024-04-15T15:23:07.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f3d3d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f41ec0d5470 con 0x7f41ec0cfda0 2024-04-15T15:23:07.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f22da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec15e910 0x7f41ec15c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:07.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f22da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec15e910 0x7f41ec15c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45660/0 (socket says 172.21.15.12:45660) 2024-04-15T15:23:07.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f22da640 1 -- 172.21.15.12:0/106629991 learned_addr learned my addr 172.21.15.12:0/106629991 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:07.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f1ad9640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41ec0b6110 0x7f41ec158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:07.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f12d8640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec0cfda0 0x7f41ec15b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:07.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f22da640 1 -- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec0cfda0 msgr2=0x7f41ec15b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:07.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f22da640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec0cfda0 0x7f41ec15b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f22da640 1 -- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41ec0b6110 msgr2=0x7f41ec158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:07.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f22da640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41ec0b6110 0x7f41ec158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.269+0000 7f41f22da640 1 -- 172.21.15.12:0/106629991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec0ce350 con 0x7f41ec15e910 2024-04-15T15:23:07.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.270+0000 7f41f22da640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec15e910 0x7f41ec15c210 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f41e400c7d0 tx=0x7f41e400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:07.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.270+0000 7f41e27fc640 1 -- 172.21.15.12:0/106629991 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4003b70 con 0x7f41ec15e910 2024-04-15T15:23:07.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.270+0000 7f41e27fc640 1 -- 172.21.15.12:0/106629991 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e4003d30 con 0x7f41ec15e910 2024-04-15T15:23:07.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.270+0000 7f41e27fc640 1 -- 172.21.15.12:0/106629991 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e400aa90 con 0x7f41ec15e910 2024-04-15T15:23:07.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.270+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec1f4dc0 con 0x7f41ec15e910 2024-04-15T15:23:07.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.270+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ec15aa00 con 0x7f41ec15e910 2024-04-15T15:23:07.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.271+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ec0689e0 con 0x7f41ec15e910 2024-04-15T15:23:07.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.272+0000 7f41e27fc640 1 -- 172.21.15.12:0/106629991 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f41e400ac30 con 0x7f41ec15e910 2024-04-15T15:23:07.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.272+0000 7f41e27fc640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f41d006c980 0x7f41d006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:07.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.272+0000 7f41e27fc640 1 -- 172.21.15.12:0/106629991 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f41e408bcc0 con 0x7f41ec15e910 2024-04-15T15:23:07.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.272+0000 7f41f1ad9640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f41d006c980 0x7f41d006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:07.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.273+0000 7f41f1ad9640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f41d006c980 0x7f41d006ee40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f41e8006fd0 tx=0x7f41e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:07.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.277+0000 7f41e27fc640 1 -- 172.21.15.12:0/106629991 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41e4059f50 con 0x7f41ec15e910 2024-04-15T15:23:07.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.499+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f41ec0b6e60 con 0x7f41ec15e910 2024-04-15T15:23:07.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.500+0000 7f41e27fc640 1 -- 172.21.15.12:0/106629991 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7f41e4059850 con 0x7f41ec15e910 2024-04-15T15:23:07.502 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:23:07.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f41d006c980 msgr2=0x7f41d006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f41d006c980 0x7f41d006ee40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f41e8006fd0 tx=0x7f41e8008040 comp rx=0 tx=0).stop 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec15e910 msgr2=0x7f41ec15c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec15e910 0x7f41ec15c210 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f41e400c7d0 tx=0x7f41e400cc90 comp rx=0 tx=0).stop 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 shutdown_connections 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41ec15e910 0x7f41ec15c210 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f41ec0cfda0 0x7f41ec15b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f41d006c980 0x7f41d006ee40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 --2- 172.21.15.12:0/106629991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41ec0b6110 0x7f41ec158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 >> 172.21.15.12:0/106629991 conn(0x7f41ec08bf60 msgr2=0x7f41ec15bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 shutdown_connections 2024-04-15T15:23:07.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07.502+0000 7f41f3d3d640 1 -- 172.21.15.12:0/106629991 wait complete. 2024-04-15T15:23:07.515 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:23:07.515 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:23:07.515 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-04-15T15:23:07.515 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:23:07.515 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:23:07.515 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:23:07.515 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a 2024-04-15T15:23:07.515 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:23:07.516 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:23:07.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:07.516 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:07.532 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42561sec preferred_lft 42561sec 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:23:07.533 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 36:f2:47:76:c7:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::18d2:66ff:fef6:52a/64 scope link 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 8e:28:9a:60:ef:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::8c28:9aff:fe60:ef2c/64 scope link 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 36:f2:47:76:c7:11 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::34f2:47ff:fe76:c711/64 scope link 2024-04-15T15:23:07.534 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:07.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:07.535 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:23:07.599 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-04-15T15:23:07.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:07.600 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-04-15T15:23:07.655 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-04-15T15:23:07.656 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:23:07.656 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:23:07.656 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:23:07.656 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:23:07.656 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:23:07.656 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:23:07.656 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:23:07.656 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:23:07.656 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:23:07.656 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:23:07.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:23:07.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:07.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:07.795 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-15T15:23:07.810 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:23:07.810 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:23:07.810 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:23:07.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:07.811 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-04-15T15:23:07.878 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:23:07.878 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:23:07.878 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:23:07.933 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:23:07.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:23:07.933 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:23:07.988 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:23:08.057 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:23:08.113 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:23:08.114 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:08.115 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:08.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:08.116 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:23:08.168 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:23:08.168 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a) 2024-04-15T15:23:08.211 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:23:08.238 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:23:08.294 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:23:08.294 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:23:08.294 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:23:08.294 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:23:08.294 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:23:08.294 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:23:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:08.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:08.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:08.297 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:23:08.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:2024-04-15T15:23:08.296+0000 7faebbe38480 -1 init, newargv = 0x565521473f60 newargc=15 2024-04-15T15:23:08.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[55627]: starting ceph client 2024-04-15T15:23:08.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[55627]: starting fuse 2024-04-15T15:23:08.313 INFO:teuthology.orchestra.run.smithi012.stdout:43 2024-04-15T15:23:08.314 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:23:08.314 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:23:08.314 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-04-15T15:23:08.314 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-04-15T15:23:08.314 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-04-15T15:23:08.314 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-04-15T15:23:08.314 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:23:08.314 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-04-15T15:23:08.315 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:23:08.316 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:23:08.316 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.0")) 2024-04-15T15:23:08.316 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-04-15T15:23:08.436 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.0.55627.asok 2024-04-15T15:23:08.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:08.446 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55627.asok 2024-04-15T15:23:08.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:08.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55627.asok status 2024-04-15T15:23:08.586 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:08.586 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-04-15T15:23:08.586 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:23:08.586 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:23:08.586 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "0", 2024-04-15T15:23:08.586 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "55627", 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a" 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5889, 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5889 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:23:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1483609703 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1483609703 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.5889 192.168.0.1:0/1483609703", 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.1:0/1483609703", 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 5, 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 18, 2024-04-15T15:23:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 41, 2024-04-15T15:23:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:23:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-04-15T15:23:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-04-15T15:23:08.589 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:08.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:23:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.697+0000 7f0ed6cd3640 1 Processor -- start 2024-04-15T15:23:08.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.697+0000 7f0ed6cd3640 1 -- start start 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.698+0000 7f0ed6cd3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed00d1770 0x7f0ed00d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.698+0000 7f0ed6cd3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed00d5190 con 0x7f0ed0161fe0 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.698+0000 7f0ed6cd3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed00cf3c0 con 0x7f0ed00b35f0 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.698+0000 7f0ed6cd3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed00cf6a0 con 0x7f0ed00d1770 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.698+0000 7f0ed4a6f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0ed0161fe0 0x7f0ed00b3fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52332/0 (socket says 172.21.15.12:52332) 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.698+0000 7f0ed4a6f640 1 -- 172.21.15.12:0/480724576 learned_addr learned my addr 172.21.15.12:0/480724576 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ed5270640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed00d1770 0x7f0ed00d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736671331 0 0) 0x7f0ed00d5190 con 0x7f0ed0161fe0 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eb8003520 con 0x7f0ed0161fe0 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 414114227 0 0) 0x7f0ed00cf3c0 con 0x7f0ed00b35f0 2024-04-15T15:23:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ed00d5190 con 0x7f0ed00b35f0 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2490295604 0 0) 0x7f0eb8003520 con 0x7f0ed0161fe0 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed00d1770 msgr2=0x7f0ed00d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed00d1770 0x7f0ed00d3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 >> v1:172.21.15.55:6789/0 conn(0x7f0ed00b35f0 legacy=0x7f0ed00d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed00cf980 con 0x7f0ed0161fe0 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f0ec80019f0 con 0x7f0ed0161fe0 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 >> v1:172.21.15.12:6789/0 conn(0x7f0ed0161fe0 legacy=0x7f0ed00b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eb8003b70 0x7f0eb8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8004800 0x7f0eb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed00d1770 0x7f0eb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed00cf6a0 con 0x7f0eb8003b70 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb800ba70 con 0x7f0eb8004800 2024-04-15T15:23:08.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb800bc70 con 0x7f0ed00d1770 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ed4a6f640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eb8003b70 0x7f0eb8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecffff640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8004800 0x7f0eb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ed5270640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed00d1770 0x7f0eb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ed4a6f640 1 -- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed00d1770 msgr2=0x7f0eb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ed4a6f640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed00d1770 0x7f0eb80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ed4a6f640 1 -- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8004800 msgr2=0x7f0eb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ed4a6f640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8004800 0x7f0eb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ed4a6f640 1 -- 172.21.15.12:0/480724576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb8003520 con 0x7f0eb8003b70 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.699+0000 7f0ecffff640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8004800 0x7f0eb8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.700+0000 7f0ed4a6f640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eb8003b70 0x7f0eb8003f70 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800a7f0 tx=0x7f0eb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a95c0e9d103ebc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.700+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec800a450 con 0x7f0eb8003b70 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.700+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec80072d0 con 0x7f0eb8003b70 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.700+0000 7f0ecf7fe640 1 -- 172.21.15.12:0/480724576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80076c0 con 0x7f0eb8003b70 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.701+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/480724576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eb8003b70 msgr2=0x7f0eb8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.701+0000 7f0ed6cd3640 1 --2- 172.21.15.12:0/480724576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0eb8003b70 0x7f0eb8003f70 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800a7f0 tx=0x7f0eb8005830 comp rx=0 tx=0).stop 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.701+0000 7f0ed5270640 1 -- 172.21.15.12:0/480724576 reap_dead start 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.701+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/480724576 shutdown_connections 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.701+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/480724576 >> 172.21.15.12:0/480724576 conn(0x7f0ed008bf60 msgr2=0x7f0ed00d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.701+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/480724576 shutdown_connections 2024-04-15T15:23:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.701+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/480724576 wait complete. 2024-04-15T15:23:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.702+0000 7f0ed6cd3640 1 Processor -- start 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.702+0000 7f0ed6cd3640 1 -- start start 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed6cd3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8003b70 0x7f0ed00cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed6cd3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed015d8e0 0x7f0ed00b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed6cd3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed00b9030 0x7f0ed00b9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed6cd3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed01623c0 con 0x7f0ed00b9030 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed6cd3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed00d5470 con 0x7f0eb8003b70 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed6cd3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed00d4d60 con 0x7f0ed015d8e0 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed5270640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed00b9030 0x7f0ed00b9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed5270640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed00b9030 0x7f0ed00b9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45678/0 (socket says 172.21.15.12:45678) 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed5270640 1 -- 172.21.15.12:0/1244461444 learned_addr learned my addr 172.21.15.12:0/1244461444 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:08.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed4a6f640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8003b70 0x7f0ed00cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ecffff640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed015d8e0 0x7f0ed00b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed5270640 1 -- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed015d8e0 msgr2=0x7f0ed00b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed5270640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed015d8e0 0x7f0ed00b71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed5270640 1 -- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8003b70 msgr2=0x7f0ed00cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed5270640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8003b70 0x7f0ed00cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.703+0000 7f0ed5270640 1 -- 172.21.15.12:0/1244461444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed00b7700 con 0x7f0ed00b9030 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.704+0000 7f0ed5270640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed00b9030 0x7f0ed00b9a50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0ed008e4c0 tx=0x7f0ec0011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.704+0000 7f0ecd7fa640 1 -- 172.21.15.12:0/1244461444 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0003610 con 0x7f0ed00b9030 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.704+0000 7f0ecd7fa640 1 -- 172.21.15.12:0/1244461444 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec001b070 con 0x7f0ed00b9030 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.704+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed00ca090 con 0x7f0ed00b9030 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.704+0000 7f0ecd7fa640 1 -- 172.21.15.12:0/1244461444 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0003a30 con 0x7f0ed00b9030 2024-04-15T15:23:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.704+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed00ca650 con 0x7f0ed00b9030 2024-04-15T15:23:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.705+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed00689e0 con 0x7f0ed00b9030 2024-04-15T15:23:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.705+0000 7f0ecd7fa640 1 -- 172.21.15.12:0/1244461444 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0ec0003bd0 con 0x7f0ed00b9030 2024-04-15T15:23:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.706+0000 7f0ecd7fa640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0ebc06c980 0x7f0ebc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.706+0000 7f0ecd7fa640 1 -- 172.21.15.12:0/1244461444 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f0ec008bab0 con 0x7f0ed00b9030 2024-04-15T15:23:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.706+0000 7f0ed4a6f640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0ebc06c980 0x7f0ebc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.706+0000 7f0ed4a6f640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0ebc06c980 0x7f0ebc06ee40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800a7f0 tx=0x7f0ec802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.710+0000 7f0ecd7fa640 1 -- 172.21.15.12:0/1244461444 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ec0024020 con 0x7f0ed00b9030 2024-04-15T15:23:08.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.933+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0ed008c470 con 0x7f0ed00b9030 2024-04-15T15:23:08.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.933+0000 7f0ecd7fa640 1 -- 172.21.15.12:0/1244461444 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7f0ec0059d40 con 0x7f0ed00b9030 2024-04-15T15:23:08.935 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:23:08.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0ebc06c980 msgr2=0x7f0ebc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:08.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0ebc06c980 0x7f0ebc06ee40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800a7f0 tx=0x7f0ec802f000 comp rx=0 tx=0).stop 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed00b9030 msgr2=0x7f0ed00b9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed00b9030 0x7f0ed00b9a50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0ed008e4c0 tx=0x7f0ec0011920 comp rx=0 tx=0).stop 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 shutdown_connections 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed00b9030 0x7f0ed00b9a50 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0ed015d8e0 0x7f0ed00b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0ebc06c980 0x7f0ebc06ee40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 --2- 172.21.15.12:0/1244461444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0eb8003b70 0x7f0ed00cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 >> 172.21.15.12:0/1244461444 conn(0x7f0ed008bf60 msgr2=0x7f0ed006d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 shutdown_connections 2024-04-15T15:23:08.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:08.935+0000 7f0ed6cd3640 1 -- 172.21.15.12:0/1244461444 wait complete. 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi055.front.sepia.ceph.com 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:23:08.948 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:23:08.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:08.949 DEBUG:teuthology.orchestra.run.smithi055:> ip addr 2024-04-15T15:23:08.968 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:23:08.968 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:23:08.968 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:23:08.968 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:08.968 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f0 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f1 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:06 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f0 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.21.15.55/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft 42600sec preferred_lft 42600sec 2024-04-15T15:23:08.969 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:07 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f1 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 2a:9f:ac:c6:f4:d0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::9cf3:7dff:fe9f:3592/64 scope link 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 2a:9f:ac:c6:f4:d0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:08.970 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::289f:acff:fec6:f4d0/64 scope link 2024-04-15T15:23:08.971 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:08.971 INFO:teuthology.orchestra.run.smithi055.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:23:08.971 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether fa:72:ab:ae:d3:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:08.971 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::f872:abff:feae:d3bd/64 scope link 2024-04-15T15:23:08.971 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:08.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:08.971 DEBUG:teuthology.orchestra.run.smithi055:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:23:09.040 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-04-15T15:23:09.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:09.042 DEBUG:teuthology.orchestra.run.smithi055:> route 2024-04-15T15:23:09.100 INFO:teuthology.orchestra.run.smithi055.stdout:Kernel IP routing table 2024-04-15T15:23:09.100 INFO:teuthology.orchestra.run.smithi055.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:23:09.100 INFO:teuthology.orchestra.run.smithi055.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:23:09.100 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:23:09.100 INFO:teuthology.orchestra.run.smithi055.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:23:09.100 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:23:09.101 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:23:09.101 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:23:09.101 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:23:09.101 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:23:09.101 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:23:09.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:23:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:23:09.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:23:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:09.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:09.248 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-04-15T15:23:09.264 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:23:09.264 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:23:09.264 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:23:09.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:09.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set brx.0 up 2024-04-15T15:23:09.333 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:23:09.333 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:23:09.333 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:23:09.388 INFO:teuthology.orchestra.run.smithi055.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:23:09.389 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:23:09.389 DEBUG:teuthology.orchestra.run.smithi055:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:23:09.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:23:09.512 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:23:09.568 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:23:09.568 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:23:09.568 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:23:09.568 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:23:09.568 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:23:09.568 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:23:09.568 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:23:09.569 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:09.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:09.571 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:23:09.623 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:23:09.623 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a) 2024-04-15T15:23:09.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:23:09.693 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:23:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:23:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:23:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:23:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:23:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:23:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:09.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:09.753 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:23:09.754 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:2024-04-15T15:23:09.752+0000 7fe11f7f4480 -1 init, newargv = 0x55e906649c20 newargc=15 2024-04-15T15:23:09.754 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:ceph-fuse[46132]: starting ceph client 2024-04-15T15:23:09.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:ceph-fuse[46132]: starting fuse 2024-04-15T15:23:09.770 INFO:teuthology.orchestra.run.smithi055.stdout:41 2024-04-15T15:23:09.770 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-04-15T15:23:09.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> import glob 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> import re 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> import os 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> import subprocess 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> def _find_admin_socket(client_name): 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> files = glob.glob(asok_path) 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> # Given a non-glob path, it better be there 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> if "*" not in asok_path: 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> assert(len(files) == 1) 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> return files[0] 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> for f in files: 2024-04-15T15:23:09.771 DEBUG:teuthology.orchestra.run.smithi055:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> contents = proc_f.read() 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> if mountpoint in contents: 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> return f 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> print(_find_admin_socket("client.1")) 2024-04-15T15:23:09.772 DEBUG:teuthology.orchestra.run.smithi055:> ' 2024-04-15T15:23:09.892 INFO:teuthology.orchestra.run.smithi055.stdout:/var/run/ceph/ceph-client.1.46132.asok 2024-04-15T15:23:09.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:23:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:09.903 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.46132.asok 2024-04-15T15:23:09.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:09.903 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.46132.asok status 2024-04-15T15:23:10.043 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-15T15:23:10.043 INFO:teuthology.orchestra.run.smithi055.stdout: "metadata": { 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "entity_id": "1", 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "hostname": "smithi055", 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "pid": "46132", 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "root": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a" 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_count": 0, 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_pinned_count": 0, 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "id": 5042, 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "inst": { 2024-04-15T15:23:10.044 INFO:teuthology.orchestra.run.smithi055.stdout: "name": { 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "client", 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "num": 5042 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 1311953379 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 1311953379 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:23:10.045 INFO:teuthology.orchestra.run.smithi055.stdout: "inst_str": "client.5042 192.168.0.1:0/1311953379", 2024-04-15T15:23:10.046 INFO:teuthology.orchestra.run.smithi055.stdout: "addr_str": "192.168.0.1:0/1311953379", 2024-04-15T15:23:10.046 INFO:teuthology.orchestra.run.smithi055.stdout: "inode_count": 5, 2024-04-15T15:23:10.046 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_epoch": 18, 2024-04-15T15:23:10.046 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch": 41, 2024-04-15T15:23:10.046 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:23:10.046 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklisted": false, 2024-04-15T15:23:10.046 INFO:teuthology.orchestra.run.smithi055.stdout: "fs_name": "cephfs" 2024-04-15T15:23:10.046 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-15T15:23:10.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:23:10.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.167+0000 7f919e90d640 1 Processor -- start 2024-04-15T15:23:10.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.167+0000 7f919e90d640 1 -- start start 2024-04-15T15:23:10.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919e90d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f91980cfda0 0x7f91980d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919e90d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91980b6110 0x7f91980b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919e90d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919815ecf0 con 0x7f919815e910 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919e90d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f91980d21d0 con 0x7f91980b6110 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919e90d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f91980d24b0 con 0x7f91980cfda0 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919ceaa640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91980b6110 0x7f91980b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f9197fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f919815e910 0x7f91980d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52346/0 (socket says 172.21.15.12:52346) 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f9197fff640 1 -- 172.21.15.12:0/1881308331 learned_addr learned my addr 172.21.15.12:0/1881308331 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f91977fe640 1 --2- 172.21.15.12:0/1881308331 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f91980cfda0 0x7f91980d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919ceaa640 1 -- 172.21.15.12:0/1881308331 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f91980cfda0 msgr2=0x7f91980d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919ceaa640 1 --2- 172.21.15.12:0/1881308331 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f91980cfda0 0x7f91980d0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919ceaa640 1 -- 172.21.15.12:0/1881308331 >> v1:172.21.15.12:6789/0 conn(0x7f919815e910 legacy=0x7f91980d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f919ceaa640 1 -- 172.21.15.12:0/1881308331 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91980d2790 con 0x7f91980b6110 2024-04-15T15:23:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.168+0000 7f9196ffd640 1 -- 172.21.15.12:0/1881308331 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2101754142 0 0) 0x7f919815ecf0 con 0x7f919815e910 2024-04-15T15:23:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.169+0000 7f919ceaa640 1 --2- 172.21.15.12:0/1881308331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980b6110 0x7f91980b65a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9190009d90 tx=0x7f919002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108055eef11276aa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.169+0000 7f9196ffd640 1 -- 172.21.15.12:0/1881308331 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190039070 con 0x7f91980b6110 2024-04-15T15:23:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.169+0000 7f9196ffd640 1 -- 172.21.15.12:0/1881308331 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919002cc50 con 0x7f91980b6110 2024-04-15T15:23:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.169+0000 7f9196ffd640 1 -- 172.21.15.12:0/1881308331 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190034440 con 0x7f91980b6110 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.170+0000 7f919e90d640 1 -- 172.21.15.12:0/1881308331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980b6110 msgr2=0x7f91980b65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.170+0000 7f919e90d640 1 --2- 172.21.15.12:0/1881308331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980b6110 0x7f91980b65a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9190009d90 tx=0x7f919002bea0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.170+0000 7f919e90d640 1 -- 172.21.15.12:0/1881308331 shutdown_connections 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.170+0000 7f919e90d640 1 --2- 172.21.15.12:0/1881308331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980b6110 0x7f91980b65a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.170+0000 7f919e90d640 1 --2- 172.21.15.12:0/1881308331 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f91980cfda0 0x7f91980d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.170+0000 7f919e90d640 1 -- 172.21.15.12:0/1881308331 >> 172.21.15.12:0/1881308331 conn(0x7f919808bf60 msgr2=0x7f91980d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.170+0000 7f919e90d640 1 -- 172.21.15.12:0/1881308331 shutdown_connections 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.170+0000 7f919e90d640 1 -- 172.21.15.12:0/1881308331 wait complete. 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.171+0000 7f919e90d640 1 Processor -- start 2024-04-15T15:23:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.171+0000 7f919e90d640 1 -- start start 2024-04-15T15:23:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f919e90d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91980b6110 0x7f919814e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f919e90d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980cfda0 0x7f91980b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f919e90d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f919815e910 0x7f9198157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f919e90d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f91980d5470 con 0x7f91980cfda0 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f919e90d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f91980d2d50 con 0x7f91980b6110 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f919e90d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f919806cc30 con 0x7f919815e910 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f91977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980cfda0 0x7f91980b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f91977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980cfda0 0x7f91980b72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45698/0 (socket says 172.21.15.12:45698) 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f91977fe640 1 -- 172.21.15.12:0/3868542381 learned_addr learned my addr 172.21.15.12:0/3868542381 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f919ceaa640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f919815e910 0x7f9198157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f9197fff640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91980b6110 0x7f919814e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f91977fe640 1 -- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f919815e910 msgr2=0x7f9198157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f91977fe640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f919815e910 0x7f9198157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f91977fe640 1 -- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91980b6110 msgr2=0x7f919814e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f91977fe640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91980b6110 0x7f919814e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.172+0000 7f91977fe640 1 -- 172.21.15.12:0/3868542381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91981f4260 con 0x7f91980cfda0 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.173+0000 7f91977fe640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980cfda0 0x7f91980b72b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f918c00d800 tx=0x7f918c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.173+0000 7f9194ff9640 1 -- 172.21.15.12:0/3868542381 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c00b660 con 0x7f91980cfda0 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.173+0000 7f9194ff9640 1 -- 172.21.15.12:0/3868542381 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918c019070 con 0x7f91980cfda0 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.173+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91980cd380 con 0x7f91980cfda0 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.173+0000 7f9194ff9640 1 -- 172.21.15.12:0/3868542381 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c00ba20 con 0x7f91980cfda0 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.173+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91980cd940 con 0x7f91980cfda0 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.174+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91980689e0 con 0x7f91980cfda0 2024-04-15T15:23:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.174+0000 7f9194ff9640 1 -- 172.21.15.12:0/3868542381 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f918c00bbc0 con 0x7f91980cfda0 2024-04-15T15:23:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.175+0000 7f9194ff9640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f918406c980 0x7f918406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.175+0000 7f9194ff9640 1 -- 172.21.15.12:0/3868542381 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f918c08c0e0 con 0x7f91980cfda0 2024-04-15T15:23:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.175+0000 7f9197fff640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f918406c980 0x7f918406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.175+0000 7f9197fff640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f918406c980 0x7f918406ee40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9188007fd0 tx=0x7f9188007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.180+0000 7f9194ff9640 1 -- 172.21.15.12:0/3868542381 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f918c059ca0 con 0x7f91980cfda0 2024-04-15T15:23:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.403+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f91980b6e60 con 0x7f91980cfda0 2024-04-15T15:23:10.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.403+0000 7f9194ff9640 1 -- 172.21.15.12:0/3868542381 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7f918c05dfa0 con 0x7f91980cfda0 2024-04-15T15:23:10.406 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:23:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f918406c980 msgr2=0x7f918406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:10.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f918406c980 0x7f918406ee40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9188007fd0 tx=0x7f9188007de0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980cfda0 msgr2=0x7f91980b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980cfda0 0x7f91980b72b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f918c00d800 tx=0x7f918c00dcc0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 shutdown_connections 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f919815e910 0x7f9198157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91980cfda0 0x7f91980b72b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f918406c980 0x7f918406ee40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 --2- 172.21.15.12:0/3868542381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91980b6110 0x7f919814e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 >> 172.21.15.12:0/3868542381 conn(0x7f919808bf60 msgr2=0x7f91980d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 shutdown_connections 2024-04-15T15:23:10.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10.405+0000 7f919e90d640 1 -- 172.21.15.12:0/3868542381 wait complete. 2024-04-15T15:23:10.418 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:23:10.418 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:23:10.418 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-04-15T15:23:10.418 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-15T15:23:10.418 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:23:10.418 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:23:10.419 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a 2024-04-15T15:23:10.419 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:23:10.419 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:23:10.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:10.421 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-04-15T15:23:10.437 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:23:10.437 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:10.438 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42558sec preferred_lft 42558sec 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 36:f2:47:76:c7:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::18d2:66ff:fef6:52a/64 scope link 2024-04-15T15:23:10.439 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run.smithi012.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 8e:28:9a:60:ef:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::8c28:9aff:fe60:ef2c/64 scope link 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run.smithi012.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 36:f2:47:76:c7:11 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::34f2:47ff:fe76:c711/64 scope link 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:10.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:10.440 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:23:10.505 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-04-15T15:23:10.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:10.506 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-04-15T15:23:10.561 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-04-15T15:23:10.562 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:23:10.562 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:23:10.562 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:23:10.562 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:23:10.562 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:23:10.562 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-15T15:23:10.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:23:10.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:23:10.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:23:10.563 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-15T15:23:10.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:23:10.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:10.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:10.701 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-15T15:23:10.717 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:23:10.717 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:23:10.717 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:23:10.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:10.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.1 up 2024-04-15T15:23:10.785 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-15T15:23:10.786 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:23:10.786 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-15T15:23:10.840 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:23:10.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:23:10.841 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:23:10.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:23:10.965 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:23:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:23:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:23:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:23:11.022 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:23:11.023 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:23:11.024 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:23:11.024 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:11.024 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:11.024 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:11.024 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:11.024 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:23:11.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:11.024 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:23:11.077 INFO:teuthology.orchestra.run.smithi012.stdout:43 2024-04-15T15:23:11.078 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:23:11.078 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a) 2024-04-15T15:23:11.120 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-04-15T15:23:11.148 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:23:11.204 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-04-15T15:23:11.204 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-04-15T15:23:11.204 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-04-15T15:23:11.204 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-04-15T15:23:11.205 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:23:11.206 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-04-15T15:23:11.207 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:11.207 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:11.207 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:11.207 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:11.207 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:23:11.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:11.207 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-04-15T15:23:11.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:2024-04-15T15:23:11.207+0000 7fab3d693480 -1 init, newargv = 0x55df4cdcff70 newargc=15 2024-04-15T15:23:11.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:ceph-fuse[56052]: starting ceph client 2024-04-15T15:23:11.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:ceph-fuse[56052]: starting fuse 2024-04-15T15:23:11.224 INFO:teuthology.orchestra.run.smithi012.stdout:43 2024-04-15T15:23:11.224 INFO:teuthology.orchestra.run.smithi012.stdout:44 2024-04-15T15:23:11.224 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-04-15T15:23:11.225 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.2")) 2024-04-15T15:23:11.226 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-04-15T15:23:11.346 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.2.56052.asok 2024-04-15T15:23:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:11.356 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.56052.asok 2024-04-15T15:23:11.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:11.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.56052.asok status 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "2", 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "56052", 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a" 2024-04-15T15:23:11.497 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5913, 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5913 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3248140896 2024-04-15T15:23:11.498 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3248140896 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.5913 192.168.0.2:0/3248140896", 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.2:0/3248140896", 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 5, 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 18, 2024-04-15T15:23:11.499 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 41, 2024-04-15T15:23:11.500 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:23:11.500 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-04-15T15:23:11.500 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-04-15T15:23:11.500 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:11.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:23:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.607+0000 7f8552f00640 1 Processor -- start 2024-04-15T15:23:11.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.607+0000 7f8552f00640 1 -- start start 2024-04-15T15:23:11.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.607+0000 7f8552f00640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f854c0b1590 0x7f854c0d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.607+0000 7f8552f00640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f854c06caf0 0x7f854c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.607+0000 7f8552f00640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c06cff0 con 0x7f854c0b1970 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f8552f00640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f854c06dc00 con 0x7f854c06caf0 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f8552f00640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f854c06d2d0 con 0x7f854c0b1590 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f854bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f854c06caf0 0x7f854c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f854bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f854c06caf0 0x7f854c09a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45708/0 (socket says 172.21.15.12:45708) 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f854bfff640 1 -- 172.21.15.12:0/190972993 learned_addr learned my addr 172.21.15.12:0/190972993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f8550c9c640 1 --2- 172.21.15.12:0/190972993 >> v2:172.21.15.55:3300/0 conn(0x7f854c0b1590 0x7f854c0d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f854bfff640 1 -- 172.21.15.12:0/190972993 >> v2:172.21.15.55:3300/0 conn(0x7f854c0b1590 msgr2=0x7f854c0d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f854bfff640 1 --2- 172.21.15.12:0/190972993 >> v2:172.21.15.55:3300/0 conn(0x7f854c0b1590 0x7f854c0d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f854bfff640 1 -- 172.21.15.12:0/190972993 >> v1:172.21.15.55:6789/0 conn(0x7f854c0b1970 legacy=0x7f854c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:11.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.608+0000 7f854bfff640 1 -- 172.21.15.12:0/190972993 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c06d5b0 con 0x7f854c06caf0 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f854bfff640 1 --2- 172.21.15.12:0/190972993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09a9d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f854c072760 tx=0x7f853802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0d0ec13455c57c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f854b7fe640 1 -- 172.21.15.12:0/190972993 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538035020 con 0x7f854c06caf0 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f854b7fe640 1 -- 172.21.15.12:0/190972993 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8538031870 con 0x7f854c06caf0 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f854b7fe640 1 -- 172.21.15.12:0/190972993 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538031c40 con 0x7f854c06caf0 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f8552f00640 1 -- 172.21.15.12:0/190972993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 msgr2=0x7f854c09a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f8552f00640 1 --2- 172.21.15.12:0/190972993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09a9d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f854c072760 tx=0x7f853802b910 comp rx=0 tx=0).stop 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f8552f00640 1 -- 172.21.15.12:0/190972993 shutdown_connections 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f8552f00640 1 --2- 172.21.15.12:0/190972993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09a9d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f8552f00640 1 --2- 172.21.15.12:0/190972993 >> v2:172.21.15.55:3300/0 conn(0x7f854c0b1590 0x7f854c0d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f8552f00640 1 -- 172.21.15.12:0/190972993 >> 172.21.15.12:0/190972993 conn(0x7f854c0b3e90 msgr2=0x7f854c0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f8552f00640 1 -- 172.21.15.12:0/190972993 shutdown_connections 2024-04-15T15:23:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.609+0000 7f8552f00640 1 -- 172.21.15.12:0/190972993 wait complete. 2024-04-15T15:23:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.611+0000 7f8552f00640 1 Processor -- start 2024-04-15T15:23:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.611+0000 7f8552f00640 1 -- start start 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.611+0000 7f8552f00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.611+0000 7f8552f00640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f854c0b1590 0x7f854c09ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.611+0000 7f8552f00640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f854c0b1970 0x7f854c09d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.611+0000 7f8552f00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f854c0d0b20 con 0x7f854c06caf0 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.611+0000 7f8552f00640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f854c0d09a0 con 0x7f854c0b1970 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.611+0000 7f8552f00640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f854c09af10 con 0x7f854c0b1590 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45714/0 (socket says 172.21.15.12:45714) 2024-04-15T15:23:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 -- 172.21.15.12:0/790652579 learned_addr learned my addr 172.21.15.12:0/790652579 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 -- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f854c0b1590 msgr2=0x7f854c09ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f854c0b1590 0x7f854c09ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 -- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f854c0b1970 msgr2=0x7f854c09d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f854c0b1970 0x7f854c09d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 -- 172.21.15.12:0/790652579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c09f430 con 0x7f854c06caf0 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8550c9c640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09c250 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f854000c7f0 tx=0x7f854000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f85497fa640 1 -- 172.21.15.12:0/790652579 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540015070 con 0x7f854c06caf0 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f85497fa640 1 -- 172.21.15.12:0/790652579 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8540004240 con 0x7f854c06caf0 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f85497fa640 1 -- 172.21.15.12:0/790652579 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540007620 con 0x7f854c06caf0 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854c09f710 con 0x7f854c06caf0 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.612+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f854c09fcd0 con 0x7f854c06caf0 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.613+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f854c0690f0 con 0x7f854c06caf0 2024-04-15T15:23:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.614+0000 7f85497fa640 1 -- 172.21.15.12:0/790652579 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f85400079d0 con 0x7f854c06caf0 2024-04-15T15:23:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.614+0000 7f85497fa640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f853006c980 0x7f853006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.614+0000 7f85497fa640 1 -- 172.21.15.12:0/790652579 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f854008bff0 con 0x7f854c06caf0 2024-04-15T15:23:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.614+0000 7f854bfff640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f853006c980 0x7f853006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.615+0000 7f854bfff640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f853006c980 0x7f853006ee40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8538036000 tx=0x7f8538030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:11.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.619+0000 7f85497fa640 1 -- 172.21.15.12:0/790652579 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f854c0690f0 con 0x7f854c06caf0 2024-04-15T15:23:11.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.845+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f854c0b8220 con 0x7f854c06caf0 2024-04-15T15:23:11.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.845+0000 7f85497fa640 1 -- 172.21.15.12:0/790652579 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7f854005a070 con 0x7f854c06caf0 2024-04-15T15:23:11.851 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:23:11.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f853006c980 msgr2=0x7f853006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:11.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f853006c980 0x7f853006ee40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8538036000 tx=0x7f8538030000 comp rx=0 tx=0).stop 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 msgr2=0x7f854c09c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09c250 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f854000c7f0 tx=0x7f854000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 shutdown_connections 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f853006c980 0x7f853006ee40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f854c0b1970 0x7f854c09d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f854c0b1590 0x7f854c09ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 --2- 172.21.15.12:0/790652579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f854c06caf0 0x7f854c09c250 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 >> 172.21.15.12:0/790652579 conn(0x7f854c0b3e90 msgr2=0x7f854c0cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 shutdown_connections 2024-04-15T15:23:11.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:11.847+0000 7f8552f00640 1 -- 172.21.15.12:0/790652579 wait complete. 2024-04-15T15:23:11.860 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:23:11.860 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:23:11.860 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi055.front.sepia.ceph.com 2024-04-15T15:23:11.860 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-15T15:23:11.861 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:23:11.861 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:23:11.861 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a 2024-04-15T15:23:11.861 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:23:11.861 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:23:11.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:11.861 DEBUG:teuthology.orchestra.run.smithi055:> ip addr 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:11.881 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f0 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f1 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:06 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f0 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.21.15.55/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft 42597sec preferred_lft 42597sec 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:07 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:11.882 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f1 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 2a:9f:ac:c6:f4:d0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::9cf3:7dff:fe9f:3592/64 scope link 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 2a:9f:ac:c6:f4:d0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::289f:acff:fec6:f4d0/64 scope link 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether fa:72:ab:ae:d3:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:11.883 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::f872:abff:feae:d3bd/64 scope link 2024-04-15T15:23:11.884 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:23:11.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:11.884 DEBUG:teuthology.orchestra.run.smithi055:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:23:11.953 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-04-15T15:23:11.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:11.955 DEBUG:teuthology.orchestra.run.smithi055:> route 2024-04-15T15:23:12.014 INFO:teuthology.orchestra.run.smithi055.stdout:Kernel IP routing table 2024-04-15T15:23:12.014 INFO:teuthology.orchestra.run.smithi055.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:23:12.014 INFO:teuthology.orchestra.run.smithi055.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:23:12.014 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:23:12.015 INFO:teuthology.orchestra.run.smithi055.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:23:12.015 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:23:12.015 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-04-15T15:23:12.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:23:12.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:23:12.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:23:12.015 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-04-15T15:23:12.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:23:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:23:12.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:23:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:12.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:12.161 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-04-15T15:23:12.177 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:23:12.177 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:23:12.177 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:23:12.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:12.178 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set brx.1 up 2024-04-15T15:23:12.247 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-15T15:23:12.247 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:23:12.247 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-15T15:23:12.302 INFO:teuthology.orchestra.run.smithi055.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:23:12.303 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:23:12.303 DEBUG:teuthology.orchestra.run.smithi055:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:23:12.358 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:23:12.426 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:23:12.483 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:23:12.483 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:23:12.483 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:23:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:23:12.485 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:23:12.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:12.486 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:23:12.538 INFO:teuthology.orchestra.run.smithi055.stdout:41 2024-04-15T15:23:12.539 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-04-15T15:23:12.539 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a) 2024-04-15T15:23:12.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-04-15T15:23:12.641 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:23:12.670 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi055.stderr:2024-04-15T15:23:12.667+0000 7f7348835480 -1 init, newargv = 0x5567f1eb4170 newargc=15 2024-04-15T15:23:12.670 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi055.stderr:ceph-fuse[46473]: starting ceph client 2024-04-15T15:23:12.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi055.stderr:ceph-fuse[46473]: starting fuse 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-04-15T15:23:12.686 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/selinux 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:23:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:12.688 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:12.689 INFO:teuthology.orchestra.run.smithi055.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:23:12.689 INFO:teuthology.orchestra.run.smithi055.stdout:/home/ubuntu/cephtest/mnt.3 2024-04-15T15:23:12.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:12.689 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-04-15T15:23:12.741 INFO:teuthology.orchestra.run.smithi055.stdout:41 2024-04-15T15:23:12.742 INFO:teuthology.orchestra.run.smithi055.stdout:44 2024-04-15T15:23:12.742 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-04-15T15:23:12.742 DEBUG:teuthology.orchestra.run.smithi055:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:23:12.742 DEBUG:teuthology.orchestra.run.smithi055:> import glob 2024-04-15T15:23:12.742 DEBUG:teuthology.orchestra.run.smithi055:> import re 2024-04-15T15:23:12.742 DEBUG:teuthology.orchestra.run.smithi055:> import os 2024-04-15T15:23:12.742 DEBUG:teuthology.orchestra.run.smithi055:> import subprocess 2024-04-15T15:23:12.742 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:23:12.742 DEBUG:teuthology.orchestra.run.smithi055:> def _find_admin_socket(client_name): 2024-04-15T15:23:12.742 DEBUG:teuthology.orchestra.run.smithi055:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> files = glob.glob(asok_path) 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> # Given a non-glob path, it better be there 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> if "*" not in asok_path: 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> assert(len(files) == 1) 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> return files[0] 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> for f in files: 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> contents = proc_f.read() 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> if mountpoint in contents: 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> return f 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:23:12.743 DEBUG:teuthology.orchestra.run.smithi055:> 2024-04-15T15:23:12.744 DEBUG:teuthology.orchestra.run.smithi055:> print(_find_admin_socket("client.3")) 2024-04-15T15:23:12.744 DEBUG:teuthology.orchestra.run.smithi055:> ' 2024-04-15T15:23:12.863 INFO:teuthology.orchestra.run.smithi055.stdout:/var/run/ceph/ceph-client.3.46473.asok 2024-04-15T15:23:12.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-15T15:23:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:12.873 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.46473.asok 2024-04-15T15:23:12.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:12.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.46473.asok status 2024-04-15T15:23:13.012 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-15T15:23:13.012 INFO:teuthology.orchestra.run.smithi055.stdout: "metadata": { 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "entity_id": "3", 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "hostname": "smithi055", 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "pid": "46473", 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "root": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a" 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_count": 0, 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_pinned_count": 0, 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "id": 5491, 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "inst": { 2024-04-15T15:23:13.013 INFO:teuthology.orchestra.run.smithi055.stdout: "name": { 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "client", 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "num": 5491 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 101850280 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:23:13.014 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 101850280 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: "inst_str": "client.5491 192.168.0.2:0/101850280", 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: "addr_str": "192.168.0.2:0/101850280", 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: "inode_count": 5, 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_epoch": 18, 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch": 41, 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklisted": false, 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout: "fs_name": "cephfs" 2024-04-15T15:23:13.015 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-15T15:23:13.021 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'touch file') 2024-04-15T15:23:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T15:23:13.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:23:13.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:13.093 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quiesce.block -v 1 file) 2024-04-15T15:23:13.112 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 quiesce path /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a --wait 2024-04-15T15:23:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.263+0000 7fe48e663640 1 Processor -- start 2024-04-15T15:23:13.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.263+0000 7fe48e663640 1 -- start start 2024-04-15T15:23:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe48e663640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe4881623f0 0x7fe4880d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe48e663640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4880d08b0 con 0x7fe48809b3e0 2024-04-15T15:23:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe48e663640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4880d0b90 con 0x7fe488162ee0 2024-04-15T15:23:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe48e663640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe4880d0680 con 0x7fe4881623f0 2024-04-15T15:23:13.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe47f7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe48809b3e0 0x7fe4881627d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47196/0 (socket says 172.21.15.12:47196) 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe47f7fe640 1 -- 172.21.15.12:0/1211624202 learned_addr learned my addr 172.21.15.12:0/1211624202 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe487fff640 1 --2- 172.21.15.12:0/1211624202 >> v2:172.21.15.55:3300/0 conn(0x7fe4881623f0 0x7fe4880d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3635275008 0 0) 0x7fe4880d08b0 con 0x7fe48809b3e0 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe474003520 con 0x7fe48809b3e0 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284310137 0 0) 0x7fe4880d0b90 con 0x7fe488162ee0 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4880d08b0 con 0x7fe488162ee0 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1791171125 0 0) 0x7fe474003520 con 0x7fe48809b3e0 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 >> v2:172.21.15.55:3300/0 conn(0x7fe4881623f0 msgr2=0x7fe4880d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 --2- 172.21.15.12:0/1211624202 >> v2:172.21.15.55:3300/0 conn(0x7fe4881623f0 0x7fe4880d1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 >> v1:172.21.15.55:6789/0 conn(0x7fe488162ee0 legacy=0x7fe48806caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4880d0e70 con 0x7fe48809b3e0 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fe470001670 con 0x7fe48809b3e0 2024-04-15T15:23:13.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 >> v1:172.21.15.12:6789/0 conn(0x7fe48809b3e0 legacy=0x7fe4881627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe474003b90 0x7fe474003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474004800 0x7fe474004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4740051c0 0x7fe47400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe47400c090 con 0x7fe474004800 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe47400c210 con 0x7fe4740051c0 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe47400c410 con 0x7fe474003b90 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.264+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474004800 0x7fe474004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.265+0000 7fe48cc00640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4740051c0 0x7fe47400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.265+0000 7fe487fff640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe474003b90 0x7fe474003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.265+0000 7fe47f7fe640 1 -- 172.21.15.12:0/1211624202 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe474003b90 msgr2=0x7fe474003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.265+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe474003b90 0x7fe474003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.265+0000 7fe47f7fe640 1 -- 172.21.15.12:0/1211624202 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4740051c0 msgr2=0x7fe47400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.265+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4740051c0 0x7fe47400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.265+0000 7fe47f7fe640 1 -- 172.21.15.12:0/1211624202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe474003520 con 0x7fe474004800 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.265+0000 7fe48cc00640 1 -- 172.21.15.12:0/1211624202 reap_dead start 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474004800 0x7fe474004c80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe470007fb0 tx=0x7fe474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81442c7540ad24d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470007c00 con 0x7fe474004800 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe470030070 con 0x7fe474004800 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe4877fe640 1 -- 172.21.15.12:0/1211624202 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470005d70 con 0x7fe474004800 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe48e663640 1 -- 172.21.15.12:0/1211624202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474004800 msgr2=0x7fe474004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe48e663640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474004800 0x7fe474004c80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe470007fb0 tx=0x7fe474005830 comp rx=0 tx=0).stop 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe48e663640 1 -- 172.21.15.12:0/1211624202 shutdown_connections 2024-04-15T15:23:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe48e663640 1 --2- 172.21.15.12:0/1211624202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474004800 0x7fe474004c80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe48e663640 1 -- 172.21.15.12:0/1211624202 >> 172.21.15.12:0/1211624202 conn(0x7fe4880b3f70 msgr2=0x7fe4880b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe48e663640 1 -- 172.21.15.12:0/1211624202 shutdown_connections 2024-04-15T15:23:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.266+0000 7fe48e663640 1 -- 172.21.15.12:0/1211624202 wait complete. 2024-04-15T15:23:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.268+0000 7fe48e663640 1 Processor -- start 2024-04-15T15:23:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.268+0000 7fe48e663640 1 -- start start 2024-04-15T15:23:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48e663640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474004800 0x7fe4881608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48e663640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4880b7940 0x7fe4881f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48e663640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4881f8260 0x7fe48815d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48e663640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4880d1670 con 0x7fe4880b7940 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48e663640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe48809b840 con 0x7fe474004800 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48e663640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe4880d17f0 con 0x7fe4881f8260 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4880b7940 0x7fe4881f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4880b7940 0x7fe4881f7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35680/0 (socket says 172.21.15.12:35680) 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 -- 172.21.15.12:0/2715909818 learned_addr learned my addr 172.21.15.12:0/2715909818 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 -- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4881f8260 msgr2=0x7fe48815d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48cc00640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4881f8260 0x7fe48815d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe487fff640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474004800 0x7fe4881608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4881f8260 0x7fe48815d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 -- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474004800 msgr2=0x7fe4881608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474004800 0x7fe4881608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 -- 172.21.15.12:0/2715909818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe48815f800 con 0x7fe4880b7940 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4880b7940 0x7fe4881f7d20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe47000c1d0 tx=0x7fe47002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe4857fa640 1 -- 172.21.15.12:0/2715909818 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47002cbd0 con 0x7fe4880b7940 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe4857fa640 1 -- 172.21.15.12:0/2715909818 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe47002cd70 con 0x7fe4880b7940 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48e663640 1 -- 172.21.15.12:0/2715909818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe48815fae0 con 0x7fe4880b7940 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe4857fa640 1 -- 172.21.15.12:0/2715909818 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470007270 con 0x7fe4880b7940 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.269+0000 7fe48e663640 1 -- 172.21.15.12:0/2715909818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4881600a0 con 0x7fe4880b7940 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.271+0000 7fe4857fa640 1 -- 172.21.15.12:0/2715909818 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe470048020 con 0x7fe4880b7940 2024-04-15T15:23:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.271+0000 7fe4857fa640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe46806c9b0 0x7fe46806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.271+0000 7fe4857fa640 1 -- 172.21.15.12:0/2715909818 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fe4700ad060 con 0x7fe4880b7940 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.271+0000 7fe487fff640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe46806c9b0 0x7fe46806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.272+0000 7fe487fff640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe46806c9b0 0x7fe46806ee70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe478009920 tx=0x7fe478008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.272+0000 7fe491251740 1 Processor -- start 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.272+0000 7fe491251740 1 -- start start 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe491251740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe491251740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b23c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe491251740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809af4360 0x562809b24150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe491251740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562809abbad0 con 0x562809b21260 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe491251740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562809a5f560 con 0x562809af4360 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe491251740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x562809a05c00 con 0x562809b20e80 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b23c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe487fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b23c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35682/0 (socket says 172.21.15.12:35682) 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe487fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:47198/0 (socket says 172.21.15.12:47198) 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 -- 172.21.15.12:0/1964535005 learned_addr learned my addr 172.21.15.12:0/1964535005 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe48cc00640 1 --2- 172.21.15.12:0/1964535005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809af4360 0x562809b24150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 -- 172.21.15.12:0/1964535005 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 msgr2=0x562809af3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/1964535005 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 -- 172.21.15.12:0/1964535005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809af4360 msgr2=0x562809b24150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/1964535005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809af4360 0x562809b24150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 -- 172.21.15.12:0/1964535005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56280983e030 con 0x562809b21260 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.273+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/1964535005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b23c10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe4700ab430 tx=0x7fe47007ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9f4e2f0b0f6811 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe47dffb640 1 -- 172.21.15.12:0/1964535005 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470083860 con 0x562809b21260 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe47dffb640 1 -- 172.21.15.12:0/1964535005 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe470030040 con 0x562809b21260 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe47dffb640 1 -- 172.21.15.12:0/1964535005 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470093170 con 0x562809b21260 2024-04-15T15:23:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 -- 172.21.15.12:0/1964535005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 msgr2=0x562809b23c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 --2- 172.21.15.12:0/1964535005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b23c10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe4700ab430 tx=0x7fe47007ab80 comp rx=0 tx=0).stop 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 -- 172.21.15.12:0/1964535005 shutdown_connections 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 --2- 172.21.15.12:0/1964535005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809af4360 0x562809b24150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 --2- 172.21.15.12:0/1964535005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b23c10 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 --2- 172.21.15.12:0/1964535005 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 -- 172.21.15.12:0/1964535005 >> 172.21.15.12:0/1964535005 conn(0x562809af6090 msgr2=0x562809b3ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 -- 172.21.15.12:0/1964535005 shutdown_connections 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.274+0000 7fe491251740 1 -- 172.21.15.12:0/1964535005 wait complete. 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.275+0000 7fe491251740 1 -- start start 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.275+0000 7fe491251740 1 Processor -- start 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe491251740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809b10190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe491251740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b25c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe491251740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b3ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe491251740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562809a19110 con 0x562809b21260 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe491251740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562809a5b520 con 0x562809b109c0 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe491251740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x562809a5f560 con 0x562809b20e80 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe491251740 10 client.0 _ll_register_callbacks cb 0x562809771410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b25c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b25c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35684/0 (socket says 172.21.15.12:35684) 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 -- 172.21.15.12:0/3947014183 learned_addr learned my addr 172.21.15.12:0/3947014183 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe487fff640 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809b10190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe48cc00640 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b3ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 -- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 msgr2=0x562809b10190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809b10190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 -- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 msgr2=0x562809b3ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b3ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 -- 172.21.15.12:0/3947014183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562809771c20 con 0x562809b21260 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe48cc00640 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b3ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b25c10 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe4700770e0 tx=0x7fe47002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27700daf3d652d56 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe466ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.276+0000 7fe466ffd640 1 -- 172.21.15.12:0/3947014183 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470098060 con 0x562809b21260 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe466ffd640 1 -- 172.21.15.12:0/3947014183 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fe470007be0 con 0x562809b21260 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 1 -- 172.21.15.12:0/3947014183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe470030040 con 0x562809b21260 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe466ffd640 1 -- 172.21.15.12:0/3947014183 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fe47009f050 con 0x562809b21260 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 10 client.5940 fetch_fsmap learned FSMap version 18 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 1 -- 172.21.15.12:0/3947014183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x562809731920 con 0x562809b21260 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe466ffd640 1 -- 172.21.15.12:0/3947014183 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe470030040 con 0x562809b21260 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe466ffd640 1 -- 172.21.15.12:0/3947014183 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7fe4700772f0 con 0x562809b21260 2024-04-15T15:23:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 10 client.5940 fetch_fsmap finished waiting for FSMap version 18 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 10 client.5940 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.d 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 20 client.5940 populate_metadata read hostname 'smithi012' 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x562809b40c90 0x562809b41070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 4 client.5940 mds_command: new command op to 5203 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe491251740 1 -- 172.21.15.12:0/3947014183 --> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562809a6cf80 con 0x562809b40c90 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe487fff640 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x562809b40c90 0x562809b41070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe487fff640 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x562809b40c90 0x562809b41070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e605b30ce84926d7 server_cookie=3ba96974eff01b51 in_seq=0 out_seq=0 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.277+0000 7fe466ffd640 10 client.5940 ms_handle_connect on v2:172.21.15.55:6835/3388147599 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe466ffd640 1 -- 172.21.15.12:0/3947014183 <== mds.0 v2:172.21.15.55:6835/3388147599 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562809a6cf80 con 0x562809b40c90 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe466ffd640 10 client.5940 handle_command_reply: tid=0 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 client.5940 shutdown 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 -- 172.21.15.12:0/3947014183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 msgr2=0x562809b25c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b25c10 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe4700770e0 tx=0x7fe47002f000 comp rx=0 tx=0).stop 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 -- 172.21.15.12:0/3947014183 shutdown_connections 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x562809b40c90 0x562809b41070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b3ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809b25c10 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe466ffd640 0 client.5940 ms_handle_reset on v2:172.21.15.55:6835/3388147599 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 --2- 172.21.15.12:0/3947014183 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809b10190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 -- 172.21.15.12:0/3947014183 >> 172.21.15.12:0/3947014183 conn(0x562809af4360 msgr2=0x562809afaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 -- 172.21.15.12:0/3947014183 shutdown_connections 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.278+0000 7fe491251740 1 -- 172.21.15.12:0/3947014183 wait complete. 2024-04-15T15:23:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.279+0000 7fe491251740 20 client.5940 trim_cache size 0 max 16384 2024-04-15T15:23:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.296+0000 7fe491251740 1 Processor -- start 2024-04-15T15:23:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.296+0000 7fe491251740 1 -- start start 2024-04-15T15:23:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe491251740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b1f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe491251740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe491251740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809af3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe491251740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4882019c0 con 0x562809b21260 2024-04-15T15:23:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe491251740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe488200aa0 con 0x562809b109c0 2024-04-15T15:23:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe491251740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe48814a9e0 con 0x562809b20e80 2024-04-15T15:23:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe48cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809af3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe48cc00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809af3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35688/0 (socket says 172.21.15.12:35688) 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe48cc00640 1 -- 172.21.15.12:0/305849564 learned_addr learned my addr 172.21.15.12:0/305849564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe487fff640 1 --2- 172.21.15.12:0/305849564 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b1f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe48cc00640 1 -- 172.21.15.12:0/305849564 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 msgr2=0x562809af2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe48cc00640 1 --2- 172.21.15.12:0/305849564 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe48cc00640 1 -- 172.21.15.12:0/305849564 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 msgr2=0x562809b1f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe48cc00640 1 --2- 172.21.15.12:0/305849564 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b1f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.297+0000 7fe48cc00640 1 -- 172.21.15.12:0/305849564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562809b27e10 con 0x562809b21260 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe48cc00640 1 --2- 172.21.15.12:0/305849564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809af3210 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe488073210 tx=0x7fe4800020a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991901fc8d413112 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe47d7fa640 1 -- 172.21.15.12:0/305849564 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480005420 con 0x562809b21260 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe47d7fa640 1 -- 172.21.15.12:0/305849564 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe48001b030 con 0x562809b21260 2024-04-15T15:23:13.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe47d7fa640 1 -- 172.21.15.12:0/305849564 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480005800 con 0x562809b21260 2024-04-15T15:23:13.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe491251740 1 -- 172.21.15.12:0/305849564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 msgr2=0x562809af3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe491251740 1 --2- 172.21.15.12:0/305849564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809af3210 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe488073210 tx=0x7fe4800020a0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe491251740 1 -- 172.21.15.12:0/305849564 shutdown_connections 2024-04-15T15:23:13.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe491251740 1 --2- 172.21.15.12:0/305849564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b21260 0x562809af3210 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe491251740 1 --2- 172.21.15.12:0/305849564 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b20e80 0x562809af2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe491251740 1 --2- 172.21.15.12:0/305849564 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b109c0 0x562809b1f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.298+0000 7fe491251740 1 -- 172.21.15.12:0/305849564 >> 172.21.15.12:0/305849564 conn(0x562809af4360 msgr2=0x562809b3dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.299+0000 7fe491251740 1 -- 172.21.15.12:0/305849564 shutdown_connections 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.299+0000 7fe491251740 1 -- 172.21.15.12:0/305849564 wait complete. 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.299+0000 7fe491251740 1 -- start start 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe491251740 1 Processor -- start 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe491251740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b109c0 0x562809b3c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe491251740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b20e80 0x562809b3c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe491251740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b21260 0x562809b3cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe491251740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562809abbad0 con 0x562809b20e80 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe491251740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe48814b030 con 0x562809b21260 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe491251740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x562809baf0b0 con 0x562809b109c0 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe491251740 10 client.0 _ll_register_callbacks cb 0x7fe4780092e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b20e80 0x562809b3c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe487fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b109c0 0x562809b3c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe48cc00640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b21260 0x562809b3cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe487fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b109c0 0x562809b3c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:47232/0 (socket says 172.21.15.12:47232) 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b20e80 0x562809b3c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35698/0 (socket says 172.21.15.12:35698) 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe487fff640 1 -- 172.21.15.12:0/3278265329 learned_addr learned my addr 172.21.15.12:0/3278265329 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.300+0000 7fe48cc00640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b21260 0x562809b3cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:50262/0 (socket says 172.21.15.12:50262) 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe487fff640 1 -- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b21260 msgr2=0x562809b3cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe487fff640 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b21260 0x562809b3cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe487fff640 1 -- 172.21.15.12:0/3278265329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b20e80 msgr2=0x562809b3c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe487fff640 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b20e80 0x562809b3c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe487fff640 1 -- 172.21.15.12:0/3278265329 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({osdmap=0}) -- 0x562809b3c5f0 con 0x562809b109c0 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b20e80 0x562809b3c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe487fff640 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b109c0 0x562809b3c0b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x562809731a10 tx=0x7fe478004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd2e9b6328fff692 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe466ffd640 10 client.0 ms_handle_connect on v2:172.21.15.55:3301/0 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe466ffd640 1 -- 172.21.15.12:0/3278265329 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47802d2d0 con 0x562809b109c0 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe466ffd640 1 -- 172.21.15.12:0/3278265329 <== mon.2 v2:172.21.15.55:3301/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fe47802d470 con 0x562809b109c0 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe491251740 1 -- 172.21.15.12:0/3278265329 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe48001b030 con 0x562809b109c0 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe466ffd640 1 -- 172.21.15.12:0/3278265329 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fe47803c050 con 0x562809b109c0 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe491251740 10 client.5072 fetch_fsmap learned FSMap version 18 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.301+0000 7fe491251740 1 -- 172.21.15.12:0/3278265329 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x562809baeac0 con 0x562809b109c0 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe466ffd640 1 -- 172.21.15.12:0/3278265329 <== mon.2 v2:172.21.15.55:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe48001b030 con 0x562809b109c0 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe466ffd640 1 -- 172.21.15.12:0/3278265329 <== mon.2 v2:172.21.15.55:3301/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7fe47803f020 con 0x562809b109c0 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe491251740 10 client.5072 fetch_fsmap finished waiting for FSMap version 18 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe491251740 10 client.5072 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.d 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe491251740 20 client.5072 populate_metadata read hostname 'smithi012' 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe491251740 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x562809b10010 0x562809b1f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe491251740 4 client.5072 mds_command: new command op to 5203 tid=0 multi_id=0 [{"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a", "wait": true}] 2024-04-15T15:23:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe491251740 1 -- 172.21.15.12:0/3278265329 --> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] -- command(tid 0: {"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a", "wait": true}) -- 0x562809afa2e0 con 0x562809b10010 2024-04-15T15:23:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x562809b10010 0x562809b1f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe47f7fe640 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x562809b10010 0x562809b1f890 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ee87c00e6a9c33d server_cookie=b5e458deed7e953a in_seq=0 out_seq=0 2024-04-15T15:23:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.302+0000 7fe466ffd640 10 client.5072 ms_handle_connect on v2:172.21.15.55:6835/3388147599 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe466ffd640 1 -- 172.21.15.12:0/3278265329 <== mds.0 v2:172.21.15.55:6835/3388147599 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2332 (crc 0 0 0) 0x562809afa2e0 con 0x562809b10010 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe466ffd640 10 client.5072 handle_command_reply: tid=0 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 client.5072 shutdown 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 -- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b109c0 msgr2=0x562809b3c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b109c0 0x562809b3c0b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x562809731a10 tx=0x7fe478004730 comp rx=0 tx=0).stop 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 -- 172.21.15.12:0/3278265329 shutdown_connections 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x562809b10010 0x562809b1f890 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562809b21260 0x562809b3cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562809b20e80 0x562809b3c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe466ffd640 0 client.5072 ms_handle_reset on v2:172.21.15.55:6835/3388147599 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 --2- 172.21.15.12:0/3278265329 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x562809b109c0 0x562809b3c0b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 -- 172.21.15.12:0/3278265329 >> 172.21.15.12:0/3278265329 conn(0x562809af4360 msgr2=0x562809b3ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 -- 172.21.15.12:0/3278265329 shutdown_connections 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 1 -- 172.21.15.12:0/3278265329 wait complete. 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.313+0000 7fe491251740 20 client.5072 trim_cache size 0 max 16384 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stdout: "op": { 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stdout: "result": 0, 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stdout: "flag_point": "quiesce complete", 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stdout: "reqid": { 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stdout: "entity": { 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "mds", 2024-04-15T15:23:13.317 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 0 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: "tid": 1 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: "op_type": "internal_op", 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: "internal_op": 5384, 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: "op_name": "quiesce_path", 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: "events": [ 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304441+0000", 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "initiated" 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.318 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304441+0000", 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "header_read" 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304441+0000", 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "throttled" 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304441+0000", 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "all_read" 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304441+0000", 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "dispatched" 2024-04-15T15:23:13.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304499+0000", 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "acquired locks" 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304519+0000", 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "acquired locks" 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304538+0000", 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "acquired locks" 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.320 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.304558+0000", 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "acquired locks" 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.314095+0000", 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "acquired locks" 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.314114+0000", 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "acquired locks" 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.314132+0000", 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "acquired locks" 2024-04-15T15:23:13.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.314152+0000", 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "acquired locks" 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: "time": "2024-04-15T15:23:13.314175+0000", 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: "event": "quiesce complete" 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: "locks": null 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: "state": { 2024-04-15T15:23:13.322 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes": 2, 2024-04-15T15:23:13.323 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_quiesced": 1, 2024-04-15T15:23:13.323 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_blocked": 1, 2024-04-15T15:23:13.323 INFO:teuthology.orchestra.run.smithi012.stdout: "inodes_dropped": 0, 2024-04-15T15:23:13.323 INFO:teuthology.orchestra.run.smithi012.stdout: "failed": [] 2024-04-15T15:23:13.323 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-15T15:23:13.323 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:13.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 -- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe46806c9b0 msgr2=0x7fe46806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe46806c9b0 0x7fe46806ee70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe478009920 tx=0x7fe478008040 comp rx=0 tx=0).stop 2024-04-15T15:23:13.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 -- 172.21.15.12:0/2715909818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4880b7940 msgr2=0x7fe4881f7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4880b7940 0x7fe4881f7d20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe47000c1d0 tx=0x7fe47002e070 comp rx=0 tx=0).stop 2024-04-15T15:23:13.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 -- 172.21.15.12:0/2715909818 shutdown_connections 2024-04-15T15:23:13.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4881f8260 0x7fe48815d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4880b7940 0x7fe4881f7d20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe46806c9b0 0x7fe46806ee70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 --2- 172.21.15.12:0/2715909818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474004800 0x7fe4881608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 -- 172.21.15.12:0/2715909818 >> 172.21.15.12:0/2715909818 conn(0x7fe4880b3f70 msgr2=0x7fe4880b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 -- 172.21.15.12:0/2715909818 shutdown_connections 2024-04-15T15:23:13.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.314+0000 7fe48e663640 1 -- 172.21.15.12:0/2715909818 wait complete. 2024-04-15T15:23:13.327 DEBUG:tasks.cephfs.test_quiesce:{'op': {'result': 0, 'flag_point': 'quiesce complete', 'reqid': {'entity': {'type': 'mds', 'num': 0}, 'tid': 1}, 'op_type': 'internal_op', 'internal_op': 5384, 'op_name': 'quiesce_path', 'events': [{'time': '2024-04-15T15:23:13.304441+0000', 'event': 'initiated'}, {'time': '2024-04-15T15:23:13.304441+0000', 'event': 'header_read'}, {'time': '2024-04-15T15:23:13.304441+0000', 'event': 'throttled'}, {'time': '2024-04-15T15:23:13.304441+0000', 'event': 'all_read'}, {'time': '2024-04-15T15:23:13.304441+0000', 'event': 'dispatched'}, {'time': '2024-04-15T15:23:13.304499+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:23:13.304519+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:23:13.304538+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:23:13.304558+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:23:13.314095+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:23:13.314114+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:23:13.314132+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:23:13.314152+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:23:13.314175+0000', 'event': 'quiesce complete'}], 'locks': None}, 'state': {'inodes': 2, 'inodes_quiesced': 1, 'inodes_blocked': 1, 'inodes_dropped': 0, 'failed': []}} 2024-04-15T15:23:13.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:23:13.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.429+0000 7f9afbec8640 1 Processor -- start 2024-04-15T15:23:13.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.430+0000 7f9afbec8640 1 -- start start 2024-04-15T15:23:13.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.430+0000 7f9afbec8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 0x7f9af40b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.430+0000 7f9afbec8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9af40b53c0 0x7f9af40b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.430+0000 7f9afbec8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9af4067e30 con 0x7f9af415ec10 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.430+0000 7f9afbec8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9af40b3c10 con 0x7f9af40b53c0 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.430+0000 7f9afbec8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9af40b3ef0 con 0x7f9af415e830 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af9463640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9af415ec10 0x7f9af4067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47200/0 (socket says 172.21.15.12:47200) 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af9463640 1 -- 172.21.15.12:0/421778379 learned_addr learned my addr 172.21.15.12:0/421778379 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af9c64640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 0x7f9af40b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9afa465640 1 --2- 172.21.15.12:0/421778379 >> v2:172.21.15.55:3300/0 conn(0x7f9af40b53c0 0x7f9af40b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 562426910 0 0) 0x7f9af4067e30 con 0x7f9af415ec10 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9adc003520 con 0x7f9af415ec10 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 791920273 0 0) 0x7f9adc003520 con 0x7f9af415ec10 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 msgr2=0x7f9af40b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 0x7f9af40b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 >> v2:172.21.15.55:3300/0 conn(0x7f9af40b53c0 msgr2=0x7f9af40b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 --2- 172.21.15.12:0/421778379 >> v2:172.21.15.55:3300/0 conn(0x7f9af40b53c0 0x7f9af40b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af40b41d0 con 0x7f9af415ec10 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f9ae8001670 con 0x7f9af415ec10 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 >> v1:172.21.15.12:6789/0 conn(0x7f9af415ec10 legacy=0x7f9af4067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 0x7f9adc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9adc003f30 0x7f9adc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9adc004d40 0x7f9adc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9af40b3c10 con 0x7f9adc004d40 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9af40b3ef0 con 0x7f9adc003f30 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9adc00b9c0 con 0x7f9af415e830 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af9c64640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 0x7f9adc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9af9463640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9adc003f30 0x7f9adc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.431+0000 7f9afa465640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9adc004d40 0x7f9adc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.432+0000 7f9af9c64640 1 -- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9adc003f30 msgr2=0x7f9adc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.432+0000 7f9af9c64640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9adc003f30 0x7f9adc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.432+0000 7f9af9c64640 1 -- 172.21.15.12:0/421778379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9adc004d40 msgr2=0x7f9adc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.432+0000 7f9af9c64640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9adc004d40 0x7f9adc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.432+0000 7f9af9c64640 1 -- 172.21.15.12:0/421778379 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9adc003520 con 0x7f9af415e830 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.432+0000 7f9afa465640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9adc004d40 0x7f9adc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.432+0000 7f9af9c64640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 0x7f9adc0039b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4009fc0 tx=0x7f9adc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb71dccaf2e987bc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4039070 con 0x7f9af415e830 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae402cc60 con 0x7f9af415e830 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9af8c62640 1 -- 172.21.15.12:0/421778379 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4009c10 con 0x7f9af415e830 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9afbec8640 1 -- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 msgr2=0x7f9adc0039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9afbec8640 1 --2- 172.21.15.12:0/421778379 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af415e830 0x7f9adc0039b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4009fc0 tx=0x7f9adc005830 comp rx=0 tx=0).stop 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9afa465640 1 -- 172.21.15.12:0/421778379 reap_dead start 2024-04-15T15:23:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9afbec8640 1 -- 172.21.15.12:0/421778379 shutdown_connections 2024-04-15T15:23:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9afbec8640 1 -- 172.21.15.12:0/421778379 >> 172.21.15.12:0/421778379 conn(0x7f9af4073360 msgr2=0x7f9af406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9afbec8640 1 -- 172.21.15.12:0/421778379 shutdown_connections 2024-04-15T15:23:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.433+0000 7f9afbec8640 1 -- 172.21.15.12:0/421778379 wait complete. 2024-04-15T15:23:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.434+0000 7f9afbec8640 1 Processor -- start 2024-04-15T15:23:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.434+0000 7f9afbec8640 1 -- start start 2024-04-15T15:23:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9afbec8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9af415e830 0x7f9af4144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9afbec8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9af41e51c0 0x7f9af4142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9afbec8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af4142d10 0x7f9af41554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9afbec8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9af415ff60 con 0x7f9af415e830 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9afbec8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9af40b4790 con 0x7f9af41e51c0 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9afbec8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9af41600e0 con 0x7f9af4142d10 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9af9c64640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9af415e830 0x7f9af4144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9af9c64640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9af415e830 0x7f9af4144720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35712/0 (socket says 172.21.15.12:35712) 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9af9c64640 1 -- 172.21.15.12:0/913887235 learned_addr learned my addr 172.21.15.12:0/913887235 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9af9463640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9af41e51c0 0x7f9af4142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.435+0000 7f9afa465640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af4142d10 0x7f9af41554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af9c64640 1 -- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af4142d10 msgr2=0x7f9af41554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af9c64640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af4142d10 0x7f9af41554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af9c64640 1 -- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9af41e51c0 msgr2=0x7f9af4142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af9c64640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9af41e51c0 0x7f9af4142650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af9c64640 1 -- 172.21.15.12:0/913887235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af4143680 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af9c64640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9af415e830 0x7f9af4144720 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400e200 tx=0x7f9ae4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af27fc640 1 -- 172.21.15.12:0/913887235 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4044030 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af27fc640 1 -- 172.21.15.12:0/913887235 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae402ca60 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af4143960 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9af27fc640 1 -- 172.21.15.12:0/913887235 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4039040 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.436+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af4143f20 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.437+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9af40689e0 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.437+0000 7f9af27fc640 1 -- 172.21.15.12:0/913887235 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9ae4047070 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.438+0000 7f9af27fc640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9ae006c9b0 0x7f9ae006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.438+0000 7f9af27fc640 1 -- 172.21.15.12:0/913887235 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f9ae40ae5f0 con 0x7f9af415e830 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.438+0000 7f9af9463640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9ae006c9b0 0x7f9ae006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.438+0000 7f9af9463640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9ae006c9b0 0x7f9ae006ee70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9af406d3c0 tx=0x7f9ae8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.442+0000 7f9af27fc640 1 -- 172.21.15.12:0/913887235 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ae407c9a0 con 0x7f9af415e830 2024-04-15T15:23:13.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.665+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9af4153010 con 0x7f9af415e830 2024-04-15T15:23:13.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.666+0000 7f9af27fc640 1 -- 172.21.15.12:0/913887235 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7f9ae407c2a0 con 0x7f9af415e830 2024-04-15T15:23:13.668 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:13.668 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:22:35.667510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6837/3388147599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":3388147599},{"type":"v1","addr":"172.21.15.55:6837","nonce":3388147599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5203,"qdb_cluster":[5203]},"id":2}]} 2024-04-15T15:23:13.669 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 18 2024-04-15T15:23:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9ae006c9b0 msgr2=0x7f9ae006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9ae006c9b0 0x7f9ae006ee70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9af406d3c0 tx=0x7f9ae8009040 comp rx=0 tx=0).stop 2024-04-15T15:23:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9af415e830 msgr2=0x7f9af4144720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9af415e830 0x7f9af4144720 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400e200 tx=0x7f9ae4009750 comp rx=0 tx=0).stop 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 shutdown_connections 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9af4142d10 0x7f9af41554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9ae006c9b0 0x7f9ae006ee70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9af41e51c0 0x7f9af4142650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 --2- 172.21.15.12:0/913887235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9af415e830 0x7f9af4144720 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 >> 172.21.15.12:0/913887235 conn(0x7f9af4073360 msgr2=0x7f9af40afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 shutdown_connections 2024-04-15T15:23:13.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.668+0000 7f9afbec8640 1 -- 172.21.15.12:0/913887235 wait complete. 2024-04-15T15:23:13.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a 0 2024-04-15T15:23:13.804 INFO:teuthology.orchestra.run.smithi055.stdout:[{"ino":1099511627779,"rdev":0,"ctime":"2024-04-15T15:23:13.080769+0000","btime":"2024-04-15T15:22:47.713864+0000","mode":16895,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":6,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-04-15T15:23:13.080769+0000","atime":"2024-04-15T15:22:47.713864+0000","time_warp_seq":0,"change_attr":2,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-04-15T15:23:13.080769+0000","num_files":1,"num_subdirs":0,"change_attr":1},"rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-15T15:23:13.108952+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-15T15:23:13.108952+0000"},"version":20,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":2,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"waiter":0,"authpin":1},"nref":8,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":1,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{"reqid":{"entity":{"type":"mds","num":0},"tid":2}}},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":5042,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5491,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5742,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":7},{"client_id":5889,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":3},{"client_id":5913,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a/file","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511629282,"remote_type":"","version":6,"projected_version":6,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-04-15T15:23:13.805 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-04-15T15:23:13.108952+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-04-15T15:22:47.713864+0000", "auth_pins": 1, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-04-15T15:22:47.713864+0000", "change_attr": 2, "client_caps": [ { "client_id": 5042, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5491, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5742, "issued": "pAsLsXs", "last_sent": 7, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5889, "issued": "pAsLsXs", "last_sent": 3, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5913, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" } ], "client_ranges": [], "ctime": "2024-04-15T15:23:13.080769+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511629282, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a/file", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 6, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 6, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 1, "mtime": "2024-04-15T15:23:13.080769+0000", "num_files": 1, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1099511627779, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 6, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16895, "mtime": "2024-04-15T15:23:13.080769+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 8, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 2, "request": 1, "waiter": 0 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "lock", "type": "iquiesce", "xlock_by": { "reqid": { "entity": { "num": 0, "type": "mds" }, "tid": 2 } } }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-04-15T15:23:13.108952+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 20, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } ] 2024-04-15T15:23:13.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 ops --flags=locks --path=/tmp/mds.0-ops 2024-04-15T15:23:13.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.912+0000 7fecc3d40640 1 Processor -- start 2024-04-15T15:23:13.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.912+0000 7fecc3d40640 1 -- start start 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc3d40640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc3d40640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecbc0d1cf0 0x7fecbc0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc3d40640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecbc15eb70 con 0x7fecbc0b7be0 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc3d40640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fecbc0d3790 con 0x7fecbc0d1cf0 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc3d40640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fecbc0d45b0 con 0x7fecbc15e790 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc22dd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecbc0d1cf0 0x7fecbc0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc22dd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fecbc0d1cf0 0x7fecbc0d4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35726/0 (socket says 172.21.15.12:35726) 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc22dd640 1 -- 172.21.15.12:0/2408896152 learned_addr learned my addr 172.21.15.12:0/2408896152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc1adc640 1 --2- 172.21.15.12:0/2408896152 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc22dd640 1 -- 172.21.15.12:0/2408896152 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 msgr2=0x7fecbc0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc22dd640 1 --2- 172.21.15.12:0/2408896152 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc22dd640 1 -- 172.21.15.12:0/2408896152 >> v1:172.21.15.55:6789/0 conn(0x7fecbc0b7be0 legacy=0x7fecbc0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc22dd640 1 -- 172.21.15.12:0/2408896152 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc0d3c80 con 0x7fecbc0d1cf0 2024-04-15T15:23:13.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.913+0000 7fecc1adc640 1 --2- 172.21.15.12:0/2408896152 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.914+0000 7fecc22dd640 1 --2- 172.21.15.12:0/2408896152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0d1cf0 0x7fecbc0d4a40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0d3910 tx=0x7fecb402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529a6a496507b113 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.914+0000 7fecc0ada640 1 -- 172.21.15.12:0/2408896152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4038070 con 0x7fecbc0d1cf0 2024-04-15T15:23:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.914+0000 7fecc0ada640 1 -- 172.21.15.12:0/2408896152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb4031700 con 0x7fecbc0d1cf0 2024-04-15T15:23:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.914+0000 7fecc0ada640 1 -- 172.21.15.12:0/2408896152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4031ad0 con 0x7fecbc0d1cf0 2024-04-15T15:23:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.915+0000 7fecc3d40640 1 -- 172.21.15.12:0/2408896152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0d1cf0 msgr2=0x7fecbc0d4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.915+0000 7fecc3d40640 1 --2- 172.21.15.12:0/2408896152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0d1cf0 0x7fecbc0d4a40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0d3910 tx=0x7fecb402b7a0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.915+0000 7fecc3d40640 1 -- 172.21.15.12:0/2408896152 shutdown_connections 2024-04-15T15:23:13.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.915+0000 7fecc3d40640 1 --2- 172.21.15.12:0/2408896152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0d1cf0 0x7fecbc0d4a40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.915+0000 7fecc3d40640 1 --2- 172.21.15.12:0/2408896152 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.915+0000 7fecc3d40640 1 -- 172.21.15.12:0/2408896152 >> 172.21.15.12:0/2408896152 conn(0x7fecbc067b40 msgr2=0x7fecbc067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.915+0000 7fecc3d40640 1 -- 172.21.15.12:0/2408896152 shutdown_connections 2024-04-15T15:23:13.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.915+0000 7fecc3d40640 1 -- 172.21.15.12:0/2408896152 wait complete. 2024-04-15T15:23:13.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.916+0000 7fecc3d40640 1 Processor -- start 2024-04-15T15:23:13.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.916+0000 7fecc3d40640 1 -- start start 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc3d40640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0b7be0 0x7fecbc1e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc3d40640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fecbc0d1cf0 0x7fecbc0b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc3d40640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc3d40640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc0d4f80 con 0x7fecbc0b7be0 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc3d40640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc0d4240 con 0x7fecbc0d1cf0 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc3d40640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fecbc0d5100 con 0x7fecbc15e790 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0b7be0 0x7fecbc1e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0b7be0 0x7fecbc1e8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35740/0 (socket says 172.21.15.12:35740) 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 -- 172.21.15.12:0/1047674006 learned_addr learned my addr 172.21.15.12:0/1047674006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc22dd640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc12db640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fecbc0d1cf0 0x7fecbc0b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 -- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 msgr2=0x7fecbc0b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 -- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fecbc0d1cf0 msgr2=0x7fecbc0b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fecbc0d1cf0 0x7fecbc0b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 -- 172.21.15.12:0/1047674006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc156090 con 0x7fecbc0b7be0 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc22dd640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc12db640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fecbc0d1cf0 0x7fecbc0b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0b7be0 0x7fecbc1e8e50 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fecb800ce30 tx=0x7fecb800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecb27fc640 1 -- 172.21.15.12:0/1047674006 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb800f890 con 0x7fecbc0b7be0 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecb27fc640 1 -- 172.21.15.12:0/1047674006 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb800fa30 con 0x7fecbc0b7be0 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecc3d40640 1 -- 172.21.15.12:0/1047674006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecbc15a6c0 con 0x7fecbc0b7be0 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.917+0000 7fecb27fc640 1 -- 172.21.15.12:0/1047674006 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb800fe00 con 0x7fecbc0b7be0 2024-04-15T15:23:13.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.918+0000 7fecc3d40640 1 -- 172.21.15.12:0/1047674006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecbc1f4780 con 0x7fecbc0b7be0 2024-04-15T15:23:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.919+0000 7fecb27fc640 1 -- 172.21.15.12:0/1047674006 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fecb8018400 con 0x7fecbc0b7be0 2024-04-15T15:23:13.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.919+0000 7fecb27fc640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feca006c980 0x7feca006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.920+0000 7fecb27fc640 1 -- 172.21.15.12:0/1047674006 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fecb808ccb0 con 0x7fecbc0b7be0 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.920+0000 7fecc12db640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feca006c980 0x7feca006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.920+0000 7fecc12db640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feca006c980 0x7feca006ee40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0b5860 tx=0x7fecac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.920+0000 7fecc6a07740 1 Processor -- start 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc6a07740 1 -- start start 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9c2da0 0x557f5b9c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9c6b60 0x557f5b9c8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc6a07740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557f5b92da80 con 0x557f5b9f28a0 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc6a07740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557f5b8d4580 con 0x557f5b9c2da0 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc6a07740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x557f5b929a40 con 0x557f5b9c6b60 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc22dd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc12db640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9c6b60 0x557f5b9c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc22dd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9f2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35742/0 (socket says 172.21.15.12:35742) 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc1adc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9c2da0 0x557f5b9c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.921+0000 7fecc22dd640 1 -- 172.21.15.12:0/2837729428 learned_addr learned my addr 172.21.15.12:0/2837729428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fecc22dd640 1 -- 172.21.15.12:0/2837729428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9c6b60 msgr2=0x557f5b9c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fecc22dd640 1 --2- 172.21.15.12:0/2837729428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9c6b60 0x557f5b9c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fecc22dd640 1 -- 172.21.15.12:0/2837729428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9c2da0 msgr2=0x557f5b9c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fecc22dd640 1 --2- 172.21.15.12:0/2837729428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9c2da0 0x557f5b9c3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fecc22dd640 1 -- 172.21.15.12:0/2837729428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f5b8e8ad0 con 0x557f5b9f28a0 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fecc22dd640 1 --2- 172.21.15.12:0/2837729428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9f2d20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009930 tx=0x7fecb4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90289a3072da134 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fec9f7fe640 1 -- 172.21.15.12:0/2837729428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4038040 con 0x557f5b9f28a0 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fec9f7fe640 1 -- 172.21.15.12:0/2837729428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb4031b20 con 0x557f5b9f28a0 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.922+0000 7fec9f7fe640 1 -- 172.21.15.12:0/2837729428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb403f460 con 0x557f5b9f28a0 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 -- 172.21.15.12:0/2837729428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 msgr2=0x557f5b9f2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 --2- 172.21.15.12:0/2837729428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9f2d20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009930 tx=0x7fecb4030000 comp rx=0 tx=0).stop 2024-04-15T15:23:13.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 -- 172.21.15.12:0/2837729428 shutdown_connections 2024-04-15T15:23:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 --2- 172.21.15.12:0/2837729428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9f2d20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 --2- 172.21.15.12:0/2837729428 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9c6b60 0x557f5b9c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 --2- 172.21.15.12:0/2837729428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9c2da0 0x557f5b9c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 -- 172.21.15.12:0/2837729428 >> 172.21.15.12:0/2837729428 conn(0x557f5b9c4e20 msgr2=0x557f5b9c6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 -- 172.21.15.12:0/2837729428 shutdown_connections 2024-04-15T15:23:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 -- 172.21.15.12:0/2837729428 wait complete. 2024-04-15T15:23:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 -- start start 2024-04-15T15:23:13.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.923+0000 7fecc6a07740 1 Processor -- start 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9e5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9e5890 0x557f5ba10600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5b9e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc6a07740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557f5b9c6ff0 con 0x557f5b9f28a0 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc6a07740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557f5b8e7940 con 0x557f5b9e5890 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc6a07740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x557f5b9c72d0 con 0x557f5b9f3f60 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc6a07740 10 client.0 _ll_register_callbacks cb 0x557f5b8e7350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc1adc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9e5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc1adc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9e5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35746/0 (socket says 172.21.15.12:35746) 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.924+0000 7fecc1adc640 1 -- 172.21.15.12:0/1506298230 learned_addr learned my addr 172.21.15.12:0/1506298230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc22dd640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5b9e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc12db640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9e5890 0x557f5ba10600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc1adc640 1 -- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 msgr2=0x557f5b9e78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5b9e78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc1adc640 1 -- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9e5890 msgr2=0x557f5ba10600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9e5890 0x557f5ba10600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc1adc640 1 -- 172.21.15.12:0/1506298230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f5b640420 con 0x557f5b9f28a0 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc22dd640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5b9e78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc12db640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9e5890 0x557f5ba10600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9e5240 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fecb800ce00 tx=0x7fecb805a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b21fe595d1fe8c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fec9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fec9cff9640 1 -- 172.21.15.12:0/1506298230 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8089060 con 0x557f5b9f28a0 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fec9cff9640 1 -- 172.21.15.12:0/1506298230 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fecb808aa60 con 0x557f5b9f28a0 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 1 -- 172.21.15.12:0/1506298230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb4031b20 con 0x557f5b9f28a0 2024-04-15T15:23:13.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fec9cff9640 1 -- 172.21.15.12:0/1506298230 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fecb8077050 con 0x557f5b9f28a0 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 10 client.5964 fetch_fsmap learned FSMap version 18 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 1 -- 172.21.15.12:0/1506298230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x557f5b70c830 con 0x557f5b9f28a0 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fec9cff9640 1 -- 172.21.15.12:0/1506298230 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb4031b20 con 0x557f5b9f28a0 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fec9cff9640 1 -- 172.21.15.12:0/1506298230 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7fecb8020030 con 0x557f5b9f28a0 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 10 client.5964 fetch_fsmap finished waiting for FSMap version 18 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 10 client.5964 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.d 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 20 client.5964 populate_metadata read hostname 'smithi012' 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x557f5b9c7e90 0x557f5b9ee430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 4 client.5964 mds_command: new command op to 5203 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.925+0000 7fecc6a07740 1 -- 172.21.15.12:0/1506298230 --> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f5b93b4a0 con 0x557f5b9c7e90 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.926+0000 7fecc12db640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x557f5b9c7e90 0x557f5b9ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.926+0000 7fec9cff9640 10 client.5964 ms_handle_connect on v2:172.21.15.55:6835/3388147599 2024-04-15T15:23:13.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.926+0000 7fecc12db640 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x557f5b9c7e90 0x557f5b9ee430 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4076e9db7edd92 server_cookie=4dfd212a13256abc in_seq=0 out_seq=0 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fec9cff9640 1 -- 172.21.15.12:0/1506298230 <== mds.0 v2:172.21.15.55:6835/3388147599 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f5b93b4a0 con 0x557f5b9c7e90 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fec9cff9640 10 client.5964 handle_command_reply: tid=0 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 client.5964 shutdown 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 -- 172.21.15.12:0/1506298230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 msgr2=0x557f5b9e5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9e5240 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fecb800ce00 tx=0x7fecb805a8b0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 -- 172.21.15.12:0/1506298230 shutdown_connections 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5b9e78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x557f5b9c7e90 0x557f5b9ee430 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9e5890 0x557f5ba10600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1506298230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5b9e5240 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fec9cff9640 0 client.5964 ms_handle_reset on v2:172.21.15.55:6835/3388147599 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 -- 172.21.15.12:0/1506298230 >> 172.21.15.12:0/1506298230 conn(0x557f5b9c2da0 msgr2=0x557f5b9f0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 -- 172.21.15.12:0/1506298230 shutdown_connections 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 1 -- 172.21.15.12:0/1506298230 wait complete. 2024-04-15T15:23:13.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.927+0000 7fecc6a07740 20 client.5964 trim_cache size 0 max 16384 2024-04-15T15:23:13.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.943+0000 7fecc6a07740 1 Processor -- start 2024-04-15T15:23:13.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.943+0000 7fecc6a07740 1 -- start start 2024-04-15T15:23:13.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9e5890 0x557f5ba0dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5ba81a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f3f60 0x557f5b9f2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc6a07740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc2012e0 con 0x557f5b9f28a0 2024-04-15T15:23:13.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc6a07740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc200300 con 0x557f5b9f3f60 2024-04-15T15:23:13.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc6a07740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fecbc0ce000 con 0x557f5b9e5890 2024-04-15T15:23:13.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5ba81a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5ba81a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35760/0 (socket says 172.21.15.12:35760) 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 -- 172.21.15.12:0/2699814204 learned_addr learned my addr 172.21.15.12:0/2699814204 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc22dd640 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f3f60 0x557f5b9f2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc1adc640 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9e5890 0x557f5ba0dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 -- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9e5890 msgr2=0x557f5ba0dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9e5890 0x557f5ba0dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 -- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f3f60 msgr2=0x557f5b9f2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f3f60 0x557f5b9f2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 -- 172.21.15.12:0/2699814204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f5b9c8f70 con 0x557f5b9f28a0 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc1adc640 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9e5890 0x557f5ba0dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc22dd640 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f3f60 0x557f5b9f2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fecc12db640 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5ba81a90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0caf90 tx=0x7fecac013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5f04c870edd38a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.944+0000 7fec9effd640 1 -- 172.21.15.12:0/2699814204 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecac014db0 con 0x557f5b9f28a0 2024-04-15T15:23:13.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fec9effd640 1 -- 172.21.15.12:0/2699814204 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecac02b070 con 0x557f5b9f28a0 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fec9effd640 1 -- 172.21.15.12:0/2699814204 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecac005270 con 0x557f5b9f28a0 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 -- 172.21.15.12:0/2699814204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 msgr2=0x557f5ba81a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5ba81a90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0caf90 tx=0x7fecac013000 comp rx=0 tx=0).stop 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 -- 172.21.15.12:0/2699814204 shutdown_connections 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f3f60 0x557f5b9f2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9f28a0 0x557f5ba81a90 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 --2- 172.21.15.12:0/2699814204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9e5890 0x557f5ba0dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 -- 172.21.15.12:0/2699814204 >> 172.21.15.12:0/2699814204 conn(0x557f5b9c2da0 msgr2=0x557f5b9e5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 -- 172.21.15.12:0/2699814204 shutdown_connections 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.945+0000 7fecc6a07740 1 -- 172.21.15.12:0/2699814204 wait complete. 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.946+0000 7fecc6a07740 1 -- start start 2024-04-15T15:23:13.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.946+0000 7fecc6a07740 1 Processor -- start 2024-04-15T15:23:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9e5890 0x557f5ba80820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f28a0 0x557f5b9ed5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5ba0f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557f5ba321d0 con 0x557f5b9e5890 2024-04-15T15:23:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1ffdb0 con 0x557f5b9f28a0 2024-04-15T15:23:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x557f5b9c5f90 con 0x557f5b9f3f60 2024-04-15T15:23:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 10 client.0 _ll_register_callbacks cb 0x7fecb800ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:23:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9e5890 0x557f5ba80820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9e5890 0x557f5ba80820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35772/0 (socket says 172.21.15.12:35772) 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 -- 172.21.15.12:0/1984476641 learned_addr learned my addr 172.21.15.12:0/1984476641 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc22dd640 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5ba0f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc12db640 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f28a0 0x557f5b9ed5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 -- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 msgr2=0x557f5ba0f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5ba0f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 -- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f28a0 msgr2=0x557f5b9ed5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f28a0 0x557f5b9ed5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 -- 172.21.15.12:0/1984476641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f5ba2efb0 con 0x557f5b9e5890 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc1adc640 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9e5890 0x557f5ba80820 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fecb808b230 tx=0x7fecb8007040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b88ccd6d673967 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fec9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fec9cff9640 1 -- 172.21.15.12:0/1984476641 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb807d020 con 0x557f5b9e5890 2024-04-15T15:23:13.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fec9cff9640 1 -- 172.21.15.12:0/1984476641 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fecb805b9f0 con 0x557f5b9e5890 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 1 -- 172.21.15.12:0/1984476641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecac02b070 con 0x557f5b9e5890 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fec9cff9640 1 -- 172.21.15.12:0/1984476641 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fecb8089050 con 0x557f5b9e5890 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 10 client.5970 fetch_fsmap learned FSMap version 18 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.947+0000 7fecc6a07740 1 -- 172.21.15.12:0/1984476641 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x557f5ba81060 con 0x557f5b9e5890 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fec9cff9640 1 -- 172.21.15.12:0/1984476641 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecac02b070 con 0x557f5b9e5890 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fec9cff9640 1 -- 172.21.15.12:0/1984476641 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7fecb8020040 con 0x557f5b9e5890 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fecc6a07740 10 client.5970 fetch_fsmap finished waiting for FSMap version 18 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fecc6a07740 10 client.5970 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.d 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fecc6a07740 20 client.5970 populate_metadata read hostname 'smithi012' 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x557f5b9ec4d0 0x557f5ba0fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fecc6a07740 4 client.5970 mds_command: new command op to 5203 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}] 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fecc6a07740 1 -- 172.21.15.12:0/1984476641 --> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}) -- 0x557f5b935310 con 0x557f5b9ec4d0 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fecc12db640 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x557f5b9ec4d0 0x557f5ba0fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fecc12db640 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x557f5b9ec4d0 0x557f5ba0fcf0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a514cd88dd32f97a server_cookie=1f3c706274fb45aa in_seq=0 out_seq=0 2024-04-15T15:23:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.948+0000 7fec9cff9640 10 client.5970 ms_handle_connect on v2:172.21.15.55:6835/3388147599 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fec9cff9640 1 -- 172.21.15.12:0/1984476641 <== mds.0 v2:172.21.15.55:6835/3388147599 1 ==== command_reply(tid 0: 0 ) ==== 8+0+33 (crc 0 0 0) 0x557f5b935310 con 0x557f5b9ec4d0 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fec9cff9640 10 client.5970 handle_command_reply: tid=0 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 client.5970 shutdown 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 -- 172.21.15.12:0/1984476641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9e5890 msgr2=0x557f5ba80820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9e5890 0x557f5ba80820 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fecb808b230 tx=0x7fecb8007040 comp rx=0 tx=0).stop 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 -- 172.21.15.12:0/1984476641 shutdown_connections 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x557f5b9f3f60 0x557f5ba0f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:6835/3388147599,v1:172.21.15.55:6837/3388147599] conn(0x557f5b9ec4d0 0x557f5ba0fcf0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557f5b9f28a0 0x557f5b9ed5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 --2- 172.21.15.12:0/1984476641 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f5b9e5890 0x557f5ba80820 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fec9cff9640 0 client.5970 ms_handle_reset on v2:172.21.15.55:6835/3388147599 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 -- 172.21.15.12:0/1984476641 >> 172.21.15.12:0/1984476641 conn(0x557f5b9c2da0 msgr2=0x557f5b9e42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 -- 172.21.15.12:0/1984476641 shutdown_connections 2024-04-15T15:23:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 1 -- 172.21.15.12:0/1984476641 wait complete. 2024-04-15T15:23:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.949+0000 7fecc6a07740 20 client.5970 trim_cache size 0 max 16384 2024-04-15T15:23:13.958 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:13.958 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/tmp/mds.0-ops" 2024-04-15T15:23:13.958 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 -- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feca006c980 msgr2=0x7feca006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feca006c980 0x7feca006ee40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0b5860 tx=0x7fecac009040 comp rx=0 tx=0).stop 2024-04-15T15:23:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 -- 172.21.15.12:0/1047674006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0b7be0 msgr2=0x7fecbc1e8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0b7be0 0x7fecbc1e8e50 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fecb800ce30 tx=0x7fecb800a650 comp rx=0 tx=0).stop 2024-04-15T15:23:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 -- 172.21.15.12:0/1047674006 shutdown_connections 2024-04-15T15:23:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fecbc15e790 0x7fecbc0b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7feca006c980 0x7feca006ee40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fecbc0d1cf0 0x7fecbc0b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 --2- 172.21.15.12:0/1047674006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fecbc0b7be0 0x7fecbc1e8e50 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 -- 172.21.15.12:0/1047674006 >> 172.21.15.12:0/1047674006 conn(0x7fecbc067b40 msgr2=0x7fecbc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 -- 172.21.15.12:0/1047674006 shutdown_connections 2024-04-15T15:23:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:13.950+0000 7fecc3d40640 1 -- 172.21.15.12:0/1047674006 wait complete. 2024-04-15T15:23:13.963 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-04-15T15:23:13.979 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.krbG5AAv5A 2024-04-15T15:23:13.979 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /tmp/mds.0-ops /tmp/tmp.krbG5AAv5A 2024-04-15T15:23:14.049 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.krbG5AAv5A 2024-04-15T15:23:14.177 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.krbG5AAv5A is 8KB 2024-04-15T15:23:14.197 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.krbG5AAv5A 2024-04-15T15:23:14.212 DEBUG:tasks.cephfs.filesystem:read 7765B of ops 2024-04-15T15:23:14.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a --path=/tmp/mds.0-cache 2024-04-15T15:23:14.371 INFO:teuthology.orchestra.run.smithi055.stdout:{"path":"/tmp/mds.0-cache"} 2024-04-15T15:23:14.371 DEBUG:tasks.cephfs.filesystem:_json_asok output { "path": "/tmp/mds.0-cache" } 2024-04-15T15:23:14.372 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-04-15T15:23:14.386 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.GTQ7q0Xv1C 2024-04-15T15:23:14.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /tmp/mds.0-cache /tmp/tmp.GTQ7q0Xv1C 2024-04-15T15:23:14.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.GTQ7q0Xv1C 2024-04-15T15:23:14.582 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.GTQ7q0Xv1C is 6KB 2024-04-15T15:23:14.630 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.GTQ7q0Xv1C 2024-04-15T15:23:14.645 DEBUG:tasks.cephfs.filesystem:read 6360B of cache 2024-04-15T15:23:14.645 INFO:tasks.cephfs.test_quiesce:qp = 1; qi = 1; qib = 1; qina = 0 2024-04-15T15:23:14.646 ERROR:tasks.cephfs.test_quiesce:inode: { "ino": 1099511627779, "rdev": 0, "ctime": "2024-04-15T15:23:13.080769+0000", "btime": "2024-04-15T15:22:47.713864+0000", "mode": 16895, "uid": 0, "gid": 0, "nlink": 1, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:23:13.080769+0000", "atime": "2024-04-15T15:22:47.713864+0000", "time_warp_seq": 0, "change_attr": 2, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": false, "client_ranges": [], "dirstat": { "version": 0, "mtime": "2024-04-15T15:23:13.080769+0000", "num_files": 1, "num_subdirs": 0, "change_attr": 1 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:23:13.108952+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:23:13.108952+0000" }, "version": 20, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 2, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5042, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5491, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5742, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 7 }, { "client_id": 5889, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 }, { "client_id": 5913, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [ { "path": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a", "dirfrag": "0x10000000003", "snapid_first": 2, "projected_version": "7", "version": "7", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "dentries": [ { "path": "volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a/file", "path_ino": 1, "snap_first": 2, "snap_last": 18446744073709551614, "is_primary": true, "is_remote": false, "is_null": false, "is_new": false, "inode": 1099511629282, "remote_type": "", "version": 6, "projected_version": 6, "auth_pins": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "is_frozen": false, "is_freezing": false, "pins": { "request": 0, "lock": 0, "inodepin": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "lock": {}, "versionlock": { "gather_set": [], "state": "lock", "type": "dversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty" ] } ] } ], "is_subvolume": false } 2024-04-15T15:23:14.646 ERROR:tasks.cephfs.test_quiesce:op: { "description": "internal op quiesce_inode:mds.0:2 fp=#0x10000000003", "initiated_at": "2024-04-15T15:23:13.304571+0000", "age": 0.64604318, "duration": 0.646174997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:23:13.304571+0000", "event": "initiated" }, { "time": "2024-04-15T15:23:13.304571+0000", "event": "header_read" }, { "time": "2024-04-15T15:23:13.304571+0000", "event": "throttled" }, { "time": "2024-04-15T15:23:13.304571+0000", "event": "all_read" }, { "time": "2024-04-15T15:23:13.304571+0000", "event": "dispatched" }, { "time": "2024-04-15T15:23:13.304743+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:23:13.305389+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:23:13.314063+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:23:13.080769+0000 1=1+0) n(v0 rc2024-04-15T15:23:13.108952+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5042=pAsLsXs/-@2,5491=pAsLsXs/-@2,5742=pAsLsXs/-@7,5889=pAsLsXs/-@3,5913=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x560642678b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "flags": 4, "wrlock_target": -1 }, { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:23:13.080769+0000 1=1+0) n(v0 rc2024-04-15T15:23:13.108952+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5042=pAsLsXs/-@2,5491=pAsLsXs/-@2,5742=pAsLsXs/-@7,5889=pAsLsXs/-@3,5913=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x560642678b00]", "lock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "flags": 2, "wrlock_target": -1 } ] } } 2024-04-15T15:23:14.646 ERROR:tasks.cephfs.test_quiesce:lock_type: iversion 2024-04-15T15:23:14.647 ERROR:tasks.cephfs.test_quiesce:cache: [ { "ino": 1099511629282, "rdev": 0, "ctime": "2024-04-15T15:23:13.108952+0000", "btime": "2024-04-15T15:23:13.080769+0000", "mode": 33188, "uid": 1000, "gid": 1301, "nlink": 1, "dir_layout": { "dir_hash": 0, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:23:13.083639+0000", "atime": "2024-04-15T15:23:13.083639+0000", "time_warp_seq": 2, "change_attr": 1, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": true, "client_ranges": [], "dirstat": { "version": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "change_attr": 0 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 0, "rsnaps": 0, "rctime": "2024-04-15T15:23:13.108952+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 0, "rsnaps": 0, "rctime": "2024-04-15T15:23:13.108952+0000" }, "version": 6, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 4, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 0, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5889, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [], "is_subvolume": false }, { "ino": 1099511627779, "rdev": 0, "ctime": "2024-04-15T15:23:13.080769+0000", "btime": "2024-04-15T15:22:47.713864+0000", "mode": 16895, "uid": 0, "gid": 0, "nlink": 1, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:23:13.080769+0000", "atime": "2024-04-15T15:22:47.713864+0000", "time_warp_seq": 0, "change_attr": 2, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": false, "client_ranges": [], "dirstat": { "version": 0, "mtime": "2024-04-15T15:23:13.080769+0000", "num_files": 1, "num_subdirs": 0, "change_attr": 1 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:23:13.108952+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:23:13.108952+0000" }, "version": 20, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 2, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5042, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5491, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5742, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 7 }, { "client_id": 5889, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 }, { "client_id": 5913, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [ { "path": "/volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a", "dirfrag": "0x10000000003", "snapid_first": 2, "projected_version": "7", "version": "7", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "dentries": [ { "path": "volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a/file", "path_ino": 1, "snap_first": 2, "snap_last": 18446744073709551614, "is_primary": true, "is_remote": false, "is_null": false, "is_new": false, "inode": 1099511629282, "remote_type": "", "version": 6, "projected_version": 6, "auth_pins": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "is_frozen": false, "is_freezing": false, "pins": { "request": 0, "lock": 0, "inodepin": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "lock": {}, "versionlock": { "gather_set": [], "state": "lock", "type": "dversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty" ] } ] } ], "is_subvolume": false } ] 2024-04-15T15:23:14.647 ERROR:tasks.cephfs.test_quiesce:ops: { "1099511627779": { "description": "internal op quiesce_inode:mds.0:2 fp=#0x10000000003", "initiated_at": "2024-04-15T15:23:13.304571+0000", "age": 0.64604318, "duration": 0.646174997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:23:13.304571+0000", "event": "initiated" }, { "time": "2024-04-15T15:23:13.304571+0000", "event": "header_read" }, { "time": "2024-04-15T15:23:13.304571+0000", "event": "throttled" }, { "time": "2024-04-15T15:23:13.304571+0000", "event": "all_read" }, { "time": "2024-04-15T15:23:13.304571+0000", "event": "dispatched" }, { "time": "2024-04-15T15:23:13.304743+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:23:13.305389+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:23:13.314063+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:23:13.080769+0000 1=1+0) n(v0 rc2024-04-15T15:23:13.108952+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5042=pAsLsXs/-@2,5491=pAsLsXs/-@2,5742=pAsLsXs/-@7,5889=pAsLsXs/-@3,5913=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x560642678b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "flags": 4, "wrlock_target": -1 }, { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/11a84459-5f7b-49d9-937c-05debaa5862a/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:23:13.080769+0000 1=1+0) n(v0 rc2024-04-15T15:23:13.108952+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5042=pAsLsXs/-@2,5491=pAsLsXs/-@2,5742=pAsLsXs/-@7,5889=pAsLsXs/-@3,5913=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x560642678b00]", "lock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "flags": 2, "wrlock_target": -1 } ] } }, "1099511629282": { "description": "internal op quiesce_inode:mds.0:3 fp=#0x100000005e2", "initiated_at": "2024-04-15T15:23:13.305643+0000", "age": 0.644971484, "duration": 0.645200779, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce blocked", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 3 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:23:13.305643+0000", "event": "initiated" }, { "time": "2024-04-15T15:23:13.305643+0000", "event": "header_read" }, { "time": "2024-04-15T15:23:13.305643+0000", "event": "throttled" }, { "time": "2024-04-15T15:23:13.305643+0000", "event": "all_read" }, { "time": "2024-04-15T15:23:13.305643+0000", "event": "dispatched" }, { "time": "2024-04-15T15:23:13.305852+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:23:13.306718+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:23:13.313832+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:23:13.313834+0000", "event": "quiesce blocked" } ], "locks": [] } } } 2024-04-15T15:23:14.648 INFO:tasks.cephfs.test_quiesce:making archive directory /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/quiesce 2024-04-15T15:23:14.650 ERROR:tasks.cephfs.test_quiesce:cache written to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/quiesce/cache9hk16o3v 2024-04-15T15:23:14.652 ERROR:tasks.cephfs.test_quiesce:ops written to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/quiesce/opsutx6zwzl 2024-04-15T15:23:14.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:23:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.764+0000 7fc4dc904640 1 Processor -- start 2024-04-15T15:23:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.764+0000 7fc4dc904640 1 -- start start 2024-04-15T15:23:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4dc904640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 0x7fc4d80b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4dc904640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4d80d2360 0x7fc4d80d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4dc904640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4d80b47a0 con 0x7fc4d815eae0 2024-04-15T15:23:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4dc904640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc4d80b53b0 con 0x7fc4d80d2360 2024-04-15T15:23:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4dc904640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d80b4a80 con 0x7fc4d815e700 2024-04-15T15:23:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4d5d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4d80d2360 0x7fc4d80d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4d5d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc4d80d2360 0x7fc4d80d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35780/0 (socket says 172.21.15.12:35780) 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4d6d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc4d815eae0 0x7fc4d80cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47202/0 (socket says 172.21.15.12:47202) 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.765+0000 7fc4d5d9b640 1 -- 172.21.15.12:0/3881728543 learned_addr learned my addr 172.21.15.12:0/3881728543 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.766+0000 7fc4d659c640 1 --2- 172.21.15.12:0/3881728543 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 0x7fc4d80b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.766+0000 7fc4d5d9b640 1 -- 172.21.15.12:0/3881728543 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 msgr2=0x7fc4d80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.766+0000 7fc4d5d9b640 1 --2- 172.21.15.12:0/3881728543 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 0x7fc4d80b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.766+0000 7fc4d5d9b640 1 -- 172.21.15.12:0/3881728543 >> v1:172.21.15.12:6789/0 conn(0x7fc4d815eae0 legacy=0x7fc4d80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.766+0000 7fc4d5d9b640 1 -- 172.21.15.12:0/3881728543 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d80b4d60 con 0x7fc4d80d2360 2024-04-15T15:23:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.766+0000 7fc4d559a640 1 -- 172.21.15.12:0/3881728543 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932825467 0 0) 0x7fc4d80b47a0 con 0x7fc4d815eae0 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.766+0000 7fc4d5d9b640 1 --2- 172.21.15.12:0/3881728543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d80d2b60 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009870 tx=0x7fc4c002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a541365a2408cd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4d559a640 1 -- 172.21.15.12:0/3881728543 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0034070 con 0x7fc4d80d2360 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4d559a640 1 -- 172.21.15.12:0/3881728543 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c0030840 con 0x7fc4d80d2360 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4d559a640 1 -- 172.21.15.12:0/3881728543 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0030c10 con 0x7fc4d80d2360 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4dc904640 1 -- 172.21.15.12:0/3881728543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 msgr2=0x7fc4d80d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4dc904640 1 --2- 172.21.15.12:0/3881728543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d80d2b60 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009870 tx=0x7fc4c002b980 comp rx=0 tx=0).stop 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4dc904640 1 -- 172.21.15.12:0/3881728543 shutdown_connections 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4dc904640 1 --2- 172.21.15.12:0/3881728543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d80d2b60 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4dc904640 1 --2- 172.21.15.12:0/3881728543 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 0x7fc4d80b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4dc904640 1 -- 172.21.15.12:0/3881728543 >> 172.21.15.12:0/3881728543 conn(0x7fc4d808bf60 msgr2=0x7fc4d80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4dc904640 1 -- 172.21.15.12:0/3881728543 shutdown_connections 2024-04-15T15:23:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.767+0000 7fc4dc904640 1 -- 172.21.15.12:0/3881728543 wait complete. 2024-04-15T15:23:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.769+0000 7fc4dc904640 1 Processor -- start 2024-04-15T15:23:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.769+0000 7fc4dc904640 1 -- start start 2024-04-15T15:23:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.769+0000 7fc4dc904640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d8152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.769+0000 7fc4dc904640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 0x7fc4d80ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.769+0000 7fc4dc904640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d815eae0 0x7fc4d815c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.769+0000 7fc4dc904640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d80b5d80 con 0x7fc4d80d2360 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.769+0000 7fc4dc904640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d80b4450 con 0x7fc4d815eae0 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.769+0000 7fc4dc904640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d80d1d40 con 0x7fc4d815e700 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d8152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d8152c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35794/0 (socket says 172.21.15.12:35794) 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 -- 172.21.15.12:0/2829281236 learned_addr learned my addr 172.21.15.12:0/2829281236 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d5d9b640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 0x7fc4d80ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d6d9d640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d815eae0 0x7fc4d815c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 -- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 msgr2=0x7fc4d80ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 0x7fc4d80ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 -- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d815eae0 msgr2=0x7fc4d815c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d815eae0 0x7fc4d815c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:14.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 -- 172.21.15.12:0/2829281236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d81f7d60 con 0x7fc4d80d2360 2024-04-15T15:23:14.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4d659c640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d8152c20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8007fa0 tx=0x7fc4c800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:14.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4ceffd640 1 -- 172.21.15.12:0/2829281236 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8014620 con 0x7fc4d80d2360 2024-04-15T15:23:14.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4ceffd640 1 -- 172.21.15.12:0/2829281236 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c8019070 con 0x7fc4d80d2360 2024-04-15T15:23:14.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d81f8040 con 0x7fc4d80d2360 2024-04-15T15:23:14.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4ceffd640 1 -- 172.21.15.12:0/2829281236 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8014a00 con 0x7fc4d80d2360 2024-04-15T15:23:14.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.770+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d81f8600 con 0x7fc4d80d2360 2024-04-15T15:23:14.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.771+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4d80689e0 con 0x7fc4d80d2360 2024-04-15T15:23:14.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.772+0000 7fc4ceffd640 1 -- 172.21.15.12:0/2829281236 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc4c8014ba0 con 0x7fc4d80d2360 2024-04-15T15:23:14.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.772+0000 7fc4ceffd640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4b806c980 0x7fc4b806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:14.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.772+0000 7fc4ceffd640 1 -- 172.21.15.12:0/2829281236 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fc4c808c6b0 con 0x7fc4d80d2360 2024-04-15T15:23:14.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.772+0000 7fc4d5d9b640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4b806c980 0x7fc4b806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:14.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.773+0000 7fc4d5d9b640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4b806c980 0x7fc4b806ee40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0036000 tx=0x7fc4c003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:14.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.777+0000 7fc4ceffd640 1 -- 172.21.15.12:0/2829281236 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc4c805a940 con 0x7fc4d80d2360 2024-04-15T15:23:15.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:14.999+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc4d80ba240 con 0x7fc4d80d2360 2024-04-15T15:23:15.002 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-04-15T15:23:15.000+0000 7fbdbcac3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:23:15.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.002+0000 7fc4ceffd640 1 -- 172.21.15.12:0/2829281236 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) ==== 161+0+0 (secure 0 0 0) 0x7fc4c805a240 con 0x7fc4d80d2360 2024-04-15T15:23:15.004 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4b806c980 msgr2=0x7fc4b806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4b806c980 0x7fc4b806ee40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0036000 tx=0x7fc4c003b040 comp rx=0 tx=0).stop 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 msgr2=0x7fc4d8152c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d8152c20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8007fa0 tx=0x7fc4c800e570 comp rx=0 tx=0).stop 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 shutdown_connections 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc4b806c980 0x7fc4b806ee40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d815eae0 0x7fc4d815c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc4d815e700 0x7fc4d80ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 --2- 172.21.15.12:0/2829281236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4d80d2360 0x7fc4d8152c20 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 >> 172.21.15.12:0/2829281236 conn(0x7fc4d808bf60 msgr2=0x7fc4d806e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 shutdown_connections 2024-04-15T15:23:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.004+0000 7fc4dc904640 1 -- 172.21.15.12:0/2829281236 wait complete. 2024-04-15T15:23:15.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-15T15:23:15.048 INFO:tasks.ceph.mds.d.smithi055.stdout:starting mds.d at 2024-04-15T15:23:15.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.124+0000 7f7d8564a640 1 Processor -- start 2024-04-15T15:23:15.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.124+0000 7f7d8564a640 1 -- start start 2024-04-15T15:23:15.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.125+0000 7f7d8564a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7d80160390 0x7f7d800d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:15.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.125+0000 7f7d8564a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800cdc20 0x7f7d8009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:15.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.125+0000 7f7d8564a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7d800ce080 0x7f7d8009af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:15.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.125+0000 7f7d8564a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8009b450 con 0x7f7d800ce080 2024-04-15T15:23:15.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.125+0000 7f7d8564a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7d800cd790 con 0x7f7d80160390 2024-04-15T15:23:15.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.125+0000 7f7d8564a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7d800d1390 con 0x7f7d800cdc20 2024-04-15T15:23:15.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7effd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7d80160390 0x7f7d800d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:15.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7e7fc640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800cdc20 0x7f7d8009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:15.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7d800ce080 0x7f7d8009af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:15.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7e7fc640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800cdc20 0x7f7d8009a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:47324/0 (socket says 172.21.15.12:47324) 2024-04-15T15:23:15.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7effd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7d80160390 0x7f7d800d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:50336/0 (socket says 172.21.15.12:50336) 2024-04-15T15:23:15.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7d800ce080 0x7f7d8009af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35806/0 (socket says 172.21.15.12:35806) 2024-04-15T15:23:15.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7effd640 1 -- 172.21.15.12:0/1769193610 learned_addr learned my addr 172.21.15.12:0/1769193610 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:15.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7e7fc640 1 -- 172.21.15.12:0/1769193610 >> v2:172.21.15.55:3300/0 conn(0x7f7d80160390 msgr2=0x7f7d800d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:15.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7e7fc640 1 --2- 172.21.15.12:0/1769193610 >> v2:172.21.15.55:3300/0 conn(0x7f7d80160390 0x7f7d800d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7e7fc640 1 -- 172.21.15.12:0/1769193610 >> v2:172.21.15.12:3300/0 conn(0x7f7d800ce080 msgr2=0x7f7d8009af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7e7fc640 1 --2- 172.21.15.12:0/1769193610 >> v2:172.21.15.12:3300/0 conn(0x7f7d800ce080 0x7f7d8009af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.126+0000 7f7d7e7fc640 1 -- 172.21.15.12:0/1769193610 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d800ccea0 con 0x7f7d800cdc20 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d7e7fc640 1 --2- 172.21.15.12:0/1769193610 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800cdc20 0x7f7d8009a9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d680097a0 tx=0x7f7d6802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c3abf0038c39b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d7dffb640 1 -- 172.21.15.12:0/1769193610 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68035020 con 0x7f7d800cdc20 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d7dffb640 1 -- 172.21.15.12:0/1769193610 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d68031810 con 0x7f7d800cdc20 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d7dffb640 1 -- 172.21.15.12:0/1769193610 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68031be0 con 0x7f7d800cdc20 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d8564a640 1 -- 172.21.15.12:0/1769193610 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800cdc20 msgr2=0x7f7d8009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d8564a640 1 --2- 172.21.15.12:0/1769193610 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800cdc20 0x7f7d8009a9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d680097a0 tx=0x7f7d6802b8b0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d8564a640 1 -- 172.21.15.12:0/1769193610 shutdown_connections 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d8564a640 1 --2- 172.21.15.12:0/1769193610 >> v2:172.21.15.12:3300/0 conn(0x7f7d800ce080 0x7f7d8009af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d8564a640 1 --2- 172.21.15.12:0/1769193610 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800cdc20 0x7f7d8009a9d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d8564a640 1 --2- 172.21.15.12:0/1769193610 >> v2:172.21.15.55:3300/0 conn(0x7f7d80160390 0x7f7d800d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.127+0000 7f7d8564a640 1 -- 172.21.15.12:0/1769193610 >> 172.21.15.12:0/1769193610 conn(0x7f7d800b3ec0 msgr2=0x7f7d800b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.128+0000 7f7d8564a640 1 -- 172.21.15.12:0/1769193610 shutdown_connections 2024-04-15T15:23:15.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.128+0000 7f7d8564a640 1 -- 172.21.15.12:0/1769193610 wait complete. 2024-04-15T15:23:15.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.129+0000 7f7d8564a640 1 Processor -- start 2024-04-15T15:23:15.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.129+0000 7f7d8564a640 1 -- start start 2024-04-15T15:23:15.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d8564a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d800cdc20 0x7f7d800c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:15.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d8564a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800ce080 0x7f7d800cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d8564a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d80160390 0x7f7d80150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d8564a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7d800cd420 con 0x7f7d80160390 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d8564a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7d800cd140 con 0x7f7d800cdc20 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d8564a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7d80160770 con 0x7f7d800ce080 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d80160390 0x7f7d80150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d800cdc20 0x7f7d800c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d80160390 0x7f7d80150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35816/0 (socket says 172.21.15.12:35816) 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7f7fe640 1 -- 172.21.15.12:0/329404101 learned_addr learned my addr 172.21.15.12:0/329404101 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7e7fc640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800ce080 0x7f7d800cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7effd640 1 -- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800ce080 msgr2=0x7f7d800cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7effd640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800ce080 0x7f7d800cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7effd640 1 -- 172.21.15.12:0/329404101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d80160390 msgr2=0x7f7d80150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7effd640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d80160390 0x7f7d80150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:15.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7effd640 1 -- 172.21.15.12:0/329404101 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d800b1800 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7f7fe640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d80160390 0x7f7d80150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d7effd640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d800cdc20 0x7f7d800c94a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800ebc0 tx=0x7f7d7800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d777fe640 1 -- 172.21.15.12:0/329404101 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7800cba0 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d777fe640 1 -- 172.21.15.12:0/329404101 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d78019070 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d800b1240 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d777fe640 1 -- 172.21.15.12:0/329404101 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78014480 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.130+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d800b5270 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.131+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d800690f0 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.132+0000 7f7d777fe640 1 -- 172.21.15.12:0/329404101 <== mon.1 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7d78005d10 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.132+0000 7f7d777fe640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d6006c980 0x7f7d6006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.132+0000 7f7d777fe640 1 -- 172.21.15.12:0/329404101 <== mon.1 v2:172.21.15.55:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f7d7808c200 con 0x7f7d800cdc20 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.133+0000 7f7d7e7fc640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d6006c980 0x7f7d6006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:15.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.133+0000 7f7d7e7fc640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d6006c980 0x7f7d6006ee40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d68036000 tx=0x7f7d68008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:15.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.137+0000 7f7d777fe640 1 -- 172.21.15.12:0/329404101 <== mon.1 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d78010040 con 0x7f7d800cdc20 2024-04-15T15:23:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:15.362+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f7d8015d9a0 con 0x7f7d800cdc20 2024-04-15T15:23:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.022+0000 7f7d777fe640 1 -- 172.21.15.12:0/329404101 <== mon.1 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v20) ==== 162+0+0 (secure 0 0 0) 0x7f7d78059c60 con 0x7f7d800cdc20 2024-04-15T15:23:16.025 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-15T15:23:16.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d6006c980 msgr2=0x7f7d6006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d6006c980 0x7f7d6006ee40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d68036000 tx=0x7f7d68008030 comp rx=0 tx=0).stop 2024-04-15T15:23:16.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d800cdc20 msgr2=0x7f7d800c94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d800cdc20 0x7f7d800c94a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800ebc0 tx=0x7f7d7800c490 comp rx=0 tx=0).stop 2024-04-15T15:23:16.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 shutdown_connections 2024-04-15T15:23:16.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d80160390 0x7f7d80150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d800ce080 0x7f7d800cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d6006c980 0x7f7d6006ee40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 --2- 172.21.15.12:0/329404101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d800cdc20 0x7f7d800c94a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 >> 172.21.15.12:0/329404101 conn(0x7f7d800b3ec0 msgr2=0x7f7d800b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:16.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 shutdown_connections 2024-04-15T15:23:16.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.025+0000 7f7d8564a640 1 -- 172.21.15.12:0/329404101 wait complete. 2024-04-15T15:23:16.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:23:16.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.152+0000 7f0c80382640 1 Processor -- start 2024-04-15T15:23:16.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.152+0000 7f0c80382640 1 -- start start 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c80382640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78070940 0x7f0c7809b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c80382640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c780d05e0 con 0x7f0c780d0da0 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c80382640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c780d1940 con 0x7f0c78160310 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c80382640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7815f720 con 0x7f0c78070940 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7e91f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0c780d0da0 0x7f0c780d1230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47212/0 (socket says 172.21.15.12:47212) 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7e91f640 1 -- 172.21.15.12:0/2929240820 learned_addr learned my addr 172.21.15.12:0/2929240820 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7e11e640 1 --2- 172.21.15.12:0/2929240820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78070940 0x7f0c7809b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7d11c640 1 -- 172.21.15.12:0/2929240820 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 361152034 0 0) 0x7f0c780d05e0 con 0x7f0c780d0da0 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7d11c640 1 -- 172.21.15.12:0/2929240820 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c64003520 con 0x7f0c780d0da0 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7d11c640 1 -- 172.21.15.12:0/2929240820 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762244899 0 0) 0x7f0c780d1940 con 0x7f0c78160310 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7d11c640 1 -- 172.21.15.12:0/2929240820 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c780d05e0 con 0x7f0c78160310 2024-04-15T15:23:16.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7e11e640 1 -- 172.21.15.12:0/2929240820 >> v1:172.21.15.55:6789/0 conn(0x7f0c78160310 legacy=0x7f0c78070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7e11e640 1 -- 172.21.15.12:0/2929240820 >> v1:172.21.15.12:6789/0 conn(0x7f0c780d0da0 legacy=0x7f0c780d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.153+0000 7f0c7e11e640 1 -- 172.21.15.12:0/2929240820 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7815fa00 con 0x7f0c78070940 2024-04-15T15:23:16.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.154+0000 7f0c7e11e640 1 --2- 172.21.15.12:0/2929240820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78070940 0x7f0c7809b880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0c740097d0 tx=0x7f0c7402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf8f7d5a6cc405c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:16.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.154+0000 7f0c7d11c640 1 -- 172.21.15.12:0/2929240820 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74035020 con 0x7f0c78070940 2024-04-15T15:23:16.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.154+0000 7f0c7d11c640 1 -- 172.21.15.12:0/2929240820 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c74031890 con 0x7f0c78070940 2024-04-15T15:23:16.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.154+0000 7f0c7d11c640 1 -- 172.21.15.12:0/2929240820 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74031c60 con 0x7f0c78070940 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.155+0000 7f0c80382640 1 -- 172.21.15.12:0/2929240820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78070940 msgr2=0x7f0c7809b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.155+0000 7f0c80382640 1 --2- 172.21.15.12:0/2929240820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78070940 0x7f0c7809b880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0c740097d0 tx=0x7f0c7402b930 comp rx=0 tx=0).stop 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.155+0000 7f0c80382640 1 -- 172.21.15.12:0/2929240820 shutdown_connections 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.155+0000 7f0c80382640 1 --2- 172.21.15.12:0/2929240820 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78070940 0x7f0c7809b880 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.155+0000 7f0c80382640 1 -- 172.21.15.12:0/2929240820 >> 172.21.15.12:0/2929240820 conn(0x7f0c780b3f40 msgr2=0x7f0c780b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.155+0000 7f0c80382640 1 -- 172.21.15.12:0/2929240820 shutdown_connections 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.155+0000 7f0c80382640 1 -- 172.21.15.12:0/2929240820 wait complete. 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.156+0000 7f0c80382640 1 Processor -- start 2024-04-15T15:23:16.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.156+0000 7f0c80382640 1 -- start start 2024-04-15T15:23:16.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c80382640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78070940 0x7f0c781f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c80382640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c780d0da0 0x7f0c7809fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c80382640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78160310 0x7f0c780c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c80382640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0c78160770 con 0x7f0c780d0da0 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c80382640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0c780d0880 con 0x7f0c78070940 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c80382640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7815f0d0 con 0x7f0c78160310 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c7d91d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c780d0da0 0x7f0c7809fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c7d91d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c780d0da0 0x7f0c7809fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35826/0 (socket says 172.21.15.12:35826) 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c7d91d640 1 -- 172.21.15.12:0/1129399529 learned_addr learned my addr 172.21.15.12:0/1129399529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c7e11e640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78070940 0x7f0c781f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.157+0000 7f0c7e91f640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78160310 0x7f0c780c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c7d91d640 1 -- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78160310 msgr2=0x7f0c780c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c7d91d640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78160310 0x7f0c780c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c7d91d640 1 -- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78070940 msgr2=0x7f0c781f4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c7d91d640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78070940 0x7f0c781f4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c7d91d640 1 -- 172.21.15.12:0/1129399529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7809c720 con 0x7f0c780d0da0 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c7d91d640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c780d0da0 0x7f0c7809fd40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800c430 tx=0x7f0c6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c6effd640 1 -- 172.21.15.12:0/1129399529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c68017070 con 0x7f0c780d0da0 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c6effd640 1 -- 172.21.15.12:0/1129399529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c680052f0 con 0x7f0c780d0da0 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7809ca00 con 0x7f0c780d0da0 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c6effd640 1 -- 172.21.15.12:0/1129399529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c68003830 con 0x7f0c780d0da0 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.158+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c7809cfc0 con 0x7f0c780d0da0 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.159+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c780689e0 con 0x7f0c780d0da0 2024-04-15T15:23:16.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.160+0000 7f0c6effd640 1 -- 172.21.15.12:0/1129399529 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0c6801a070 con 0x7f0c780d0da0 2024-04-15T15:23:16.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.160+0000 7f0c6effd640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c5c06c980 0x7f0c5c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:16.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.160+0000 7f0c6effd640 1 -- 172.21.15.12:0/1129399529 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0c6808b180 con 0x7f0c780d0da0 2024-04-15T15:23:16.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.160+0000 7f0c7e11e640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c5c06c980 0x7f0c5c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:16.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.161+0000 7f0c7e11e640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c5c06c980 0x7f0c5c06ee40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0c74036000 tx=0x7f0c7403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:16.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.165+0000 7f0c6effd640 1 -- 172.21.15.12:0/1129399529 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c68058c60 con 0x7f0c780d0da0 2024-04-15T15:23:16.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.386+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c78070720 con 0x7f0c780d0da0 2024-04-15T15:23:16.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.386+0000 7f0c6effd640 1 -- 172.21.15.12:0/1129399529 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5172 (secure 0 0 0) 0x7f0c6805cf60 con 0x7f0c780d0da0 2024-04-15T15:23:16.389 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:16.389 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5102,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/852515692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":852515692},{"type":"v1","addr":"172.21.15.55:6837","nonce":852515692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:23:16.019011+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5200":{"gid":5200,"name":"b","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-15T15:23:16.389 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 20 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c5c06c980 msgr2=0x7f0c5c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c5c06c980 0x7f0c5c06ee40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0c74036000 tx=0x7f0c7403d040 comp rx=0 tx=0).stop 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c780d0da0 msgr2=0x7f0c7809fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c780d0da0 0x7f0c7809fd40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800c430 tx=0x7f0c6800c8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 shutdown_connections 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0c78160310 0x7f0c780c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c780d0da0 0x7f0c7809fd40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0c5c06c980 0x7f0c5c06ee40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 --2- 172.21.15.12:0/1129399529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78070940 0x7f0c781f4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:16.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 >> 172.21.15.12:0/1129399529 conn(0x7f0c780b3f40 msgr2=0x7f0c780b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:16.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 shutdown_connections 2024-04-15T15:23:16.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:16.388+0000 7f0c80382640 1 -- 172.21.15.12:0/1129399529 wait complete. 2024-04-15T15:23:16.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:22:32.656644+0000', 'modified': '2024-04-15T15:23:16.019011+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5200}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5200': {'gid': 5200, 'name': 'b', 'rank': 0, 'incarnation': 20, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.55:6836/1202155349', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 1202155349}, {'type': 'v1', 'addr': '172.21.15.55:6836', 'nonce': 1202155349}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-15T15:23:16.805 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5200:up:replay 2024-04-15T15:23:17.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:23:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.922+0000 7f8f45b75640 1 Processor -- start 2024-04-15T15:23:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.922+0000 7f8f45b75640 1 -- start start 2024-04-15T15:23:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.922+0000 7f8f45b75640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f4015c940 0x7f8f400b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.922+0000 7f8f45b75640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f400b47f0 con 0x7f8f400b4270 2024-04-15T15:23:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.922+0000 7f8f45b75640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f4015be60 con 0x7f8f40155ce0 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.922+0000 7f8f45b75640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4015c4b0 con 0x7f8f4015c940 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8f400b4270 0x7f8f400ae9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47218/0 (socket says 172.21.15.12:47218) 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3ffff640 1 -- 172.21.15.12:0/2728546204 learned_addr learned my addr 172.21.15.12:0/2728546204 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3effd640 1 --2- 172.21.15.12:0/2728546204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f4015c940 0x7f8f400b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3e7fc640 1 -- 172.21.15.12:0/2728546204 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3466579575 0 0) 0x7f8f400b47f0 con 0x7f8f400b4270 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3e7fc640 1 -- 172.21.15.12:0/2728546204 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f28003520 con 0x7f8f400b4270 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3e7fc640 1 -- 172.21.15.12:0/2728546204 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2665496173 0 0) 0x7f8f4015be60 con 0x7f8f40155ce0 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3e7fc640 1 -- 172.21.15.12:0/2728546204 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f400b47f0 con 0x7f8f40155ce0 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3effd640 1 -- 172.21.15.12:0/2728546204 >> v1:172.21.15.55:6789/0 conn(0x7f8f40155ce0 legacy=0x7f8f4015fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3effd640 1 -- 172.21.15.12:0/2728546204 >> v1:172.21.15.12:6789/0 conn(0x7f8f400b4270 legacy=0x7f8f400ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.923+0000 7f8f3effd640 1 -- 172.21.15.12:0/2728546204 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4015c140 con 0x7f8f4015c940 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.924+0000 7f8f3effd640 1 --2- 172.21.15.12:0/2728546204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f4015c940 0x7f8f400b4d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8f340097d0 tx=0x7f8f3402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a9be6a61850346 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.924+0000 7f8f3e7fc640 1 -- 172.21.15.12:0/2728546204 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34035020 con 0x7f8f4015c940 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.924+0000 7f8f3e7fc640 1 -- 172.21.15.12:0/2728546204 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f34031800 con 0x7f8f4015c940 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.924+0000 7f8f3e7fc640 1 -- 172.21.15.12:0/2728546204 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34031bd0 con 0x7f8f4015c940 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.925+0000 7f8f45b75640 1 -- 172.21.15.12:0/2728546204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f4015c940 msgr2=0x7f8f400b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.925+0000 7f8f45b75640 1 --2- 172.21.15.12:0/2728546204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f4015c940 0x7f8f400b4d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8f340097d0 tx=0x7f8f3402b8a0 comp rx=0 tx=0).stop 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.925+0000 7f8f45b75640 1 -- 172.21.15.12:0/2728546204 shutdown_connections 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.925+0000 7f8f45b75640 1 --2- 172.21.15.12:0/2728546204 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f4015c940 0x7f8f400b4d80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.925+0000 7f8f45b75640 1 -- 172.21.15.12:0/2728546204 >> 172.21.15.12:0/2728546204 conn(0x7f8f400b6790 msgr2=0x7f8f400b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.925+0000 7f8f45b75640 1 -- 172.21.15.12:0/2728546204 shutdown_connections 2024-04-15T15:23:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.925+0000 7f8f45b75640 1 -- 172.21.15.12:0/2728546204 wait complete. 2024-04-15T15:23:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.926+0000 7f8f45b75640 1 Processor -- start 2024-04-15T15:23:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.926+0000 7f8f45b75640 1 -- start start 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f45b75640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f400b4270 0x7f8f400b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f45b75640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f40155ce0 0x7f8f40148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f45b75640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f4015c940 0x7f8f40148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f45b75640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f400b5440 con 0x7f8f4015c940 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f45b75640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8f400b52c0 con 0x7f8f400b4270 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f45b75640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8f400b55c0 con 0x7f8f40155ce0 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f3ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f4015c940 0x7f8f40148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f3ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f4015c940 0x7f8f40148f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35828/0 (socket says 172.21.15.12:35828) 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f3ffff640 1 -- 172.21.15.12:0/3668772399 learned_addr learned my addr 172.21.15.12:0/3668772399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f3effd640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f40155ce0 0x7f8f40148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.927+0000 7f8f3f7fe640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f400b4270 0x7f8f400b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f3ffff640 1 -- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f40155ce0 msgr2=0x7f8f40148a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f3ffff640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f40155ce0 0x7f8f40148a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f3ffff640 1 -- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f400b4270 msgr2=0x7f8f400b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f3ffff640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f400b4270 0x7f8f400b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f3ffff640 1 -- 172.21.15.12:0/3668772399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f40157190 con 0x7f8f4015c940 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f3f7fe640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f400b4270 0x7f8f400b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f3ffff640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f4015c940 0x7f8f40148f50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800cb20 tx=0x7f8f3800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f27fff640 1 -- 172.21.15.12:0/3668772399 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f38017070 con 0x7f8f4015c940 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f27fff640 1 -- 172.21.15.12:0/3668772399 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f3800aca0 con 0x7f8f4015c940 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f27fff640 1 -- 172.21.15.12:0/3668772399 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f38012600 con 0x7f8f4015c940 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f40157470 con 0x7f8f4015c940 2024-04-15T15:23:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.928+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f40157d10 con 0x7f8f4015c940 2024-04-15T15:23:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.929+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f40157860 con 0x7f8f4015c940 2024-04-15T15:23:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.930+0000 7f8f27fff640 1 -- 172.21.15.12:0/3668772399 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8f3801a070 con 0x7f8f4015c940 2024-04-15T15:23:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.930+0000 7f8f27fff640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8f2c06c980 0x7f8f2c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.930+0000 7f8f27fff640 1 -- 172.21.15.12:0/3668772399 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f8f3808ba50 con 0x7f8f4015c940 2024-04-15T15:23:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.930+0000 7f8f3f7fe640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8f2c06c980 0x7f8f2c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.931+0000 7f8f3f7fe640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8f2c06c980 0x7f8f2c06ee40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8f30001710 tx=0x7f8f30009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:17.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:17.935+0000 7f8f27fff640 1 -- 172.21.15.12:0/3668772399 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f38059640 con 0x7f8f4015c940 2024-04-15T15:23:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.164+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f400689e0 con 0x7f8f4015c940 2024-04-15T15:23:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.165+0000 7f8f27fff640 1 -- 172.21.15.12:0/3668772399 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5173 (secure 0 0 0) 0x7f8f3805d940 con 0x7f8f4015c940 2024-04-15T15:23:18.167 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:18.167 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5102,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/852515692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":852515692},{"type":"v1","addr":"172.21.15.55:6837","nonce":852515692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:23:17.040072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5200":{"gid":5200,"name":"b","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-15T15:23:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 22 2024-04-15T15:23:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8f2c06c980 msgr2=0x7f8f2c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8f2c06c980 0x7f8f2c06ee40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8f30001710 tx=0x7f8f30009040 comp rx=0 tx=0).stop 2024-04-15T15:23:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f4015c940 msgr2=0x7f8f40148f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f4015c940 0x7f8f40148f50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800cb20 tx=0x7f8f3800a060 comp rx=0 tx=0).stop 2024-04-15T15:23:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 shutdown_connections 2024-04-15T15:23:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f4015c940 0x7f8f40148f50 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8f40155ce0 0x7f8f40148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8f2c06c980 0x7f8f2c06ee40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 --2- 172.21.15.12:0/3668772399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f400b4270 0x7f8f400b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 >> 172.21.15.12:0/3668772399 conn(0x7f8f400b6790 msgr2=0x7f8f400b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 shutdown_connections 2024-04-15T15:23:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:18.167+0000 7f8f45b75640 1 -- 172.21.15.12:0/3668772399 wait complete. 2024-04-15T15:23:18.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:22:32.656644+0000', 'modified': '2024-04-15T15:23:17.040072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5200}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5200': {'gid': 5200, 'name': 'b', 'rank': 0, 'incarnation': 20, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.55:6836/1202155349', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 1202155349}, {'type': 'v1', 'addr': '172.21.15.55:6836', 'nonce': 1202155349}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-15T15:23:18.582 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5200:up:rejoin 2024-04-15T15:23:19.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:23:19.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.699+0000 7f51d5b24640 1 Processor -- start 2024-04-15T15:23:19.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.699+0000 7f51d5b24640 1 -- start start 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51d5b24640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d0072d80 0x7f51d0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51d5b24640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f51d00af3e0 0x7f51d00af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51d5b24640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51d015db40 con 0x7f51d00d0690 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51d5b24640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f51d015cfc0 con 0x7f51d00af3e0 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51d5b24640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f51d015d2a0 con 0x7f51d0072d80 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51ceffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f51d00af3e0 0x7f51d00af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51ceffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f51d00af3e0 0x7f51d00af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35842/0 (socket says 172.21.15.12:35842) 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51ceffd640 1 -- 172.21.15.12:0/2793768586 learned_addr learned my addr 172.21.15.12:0/2793768586 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:19.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51cf7fe640 1 --2- 172.21.15.12:0/2793768586 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d0072d80 0x7f51d0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:19.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51ceffd640 1 -- 172.21.15.12:0/2793768586 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d0072d80 msgr2=0x7f51d0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:19.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51ceffd640 1 --2- 172.21.15.12:0/2793768586 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d0072d80 0x7f51d0071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51ceffd640 1 -- 172.21.15.12:0/2793768586 >> v1:172.21.15.55:6789/0 conn(0x7f51d00d0690 legacy=0x7f51d006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:19.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.700+0000 7f51ceffd640 1 -- 172.21.15.12:0/2793768586 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d015d580 con 0x7f51d00af3e0 2024-04-15T15:23:19.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.701+0000 7f51ceffd640 1 --2- 172.21.15.12:0/2793768586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d00af840 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009870 tx=0x7f51c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c595c3d3c08330cc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:19.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.701+0000 7f51ce7fc640 1 -- 172.21.15.12:0/2793768586 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4035020 con 0x7f51d00af3e0 2024-04-15T15:23:19.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.701+0000 7f51ce7fc640 1 -- 172.21.15.12:0/2793768586 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c4031640 con 0x7f51d00af3e0 2024-04-15T15:23:19.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.701+0000 7f51ce7fc640 1 -- 172.21.15.12:0/2793768586 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c402d040 con 0x7f51d00af3e0 2024-04-15T15:23:19.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.702+0000 7f51d5b24640 1 -- 172.21.15.12:0/2793768586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 msgr2=0x7f51d00af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:19.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.702+0000 7f51d5b24640 1 --2- 172.21.15.12:0/2793768586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d00af840 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009870 tx=0x7f51c402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.702+0000 7f51d5b24640 1 -- 172.21.15.12:0/2793768586 shutdown_connections 2024-04-15T15:23:19.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.702+0000 7f51d5b24640 1 --2- 172.21.15.12:0/2793768586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d00af840 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.702+0000 7f51d5b24640 1 --2- 172.21.15.12:0/2793768586 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d0072d80 0x7f51d0071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.702+0000 7f51d5b24640 1 -- 172.21.15.12:0/2793768586 >> 172.21.15.12:0/2793768586 conn(0x7f51d006be00 msgr2=0x7f51d006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:19.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.702+0000 7f51d5b24640 1 -- 172.21.15.12:0/2793768586 shutdown_connections 2024-04-15T15:23:19.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.702+0000 7f51d5b24640 1 -- 172.21.15.12:0/2793768586 wait complete. 2024-04-15T15:23:19.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.703+0000 7f51d5b24640 1 Processor -- start 2024-04-15T15:23:19.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.703+0000 7f51d5b24640 1 -- start start 2024-04-15T15:23:19.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51d5b24640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51d0072d80 0x7f51d0146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:19.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51d5b24640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d01f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51d5b24640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d00d0690 0x7f51d01f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51d5b24640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51d015cd10 con 0x7f51d00af3e0 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51d5b24640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51d015cb90 con 0x7f51d0072d80 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51d5b24640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f51d00d0b20 con 0x7f51d00d0690 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51ceffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d01f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51ceffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d01f2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35844/0 (socket says 172.21.15.12:35844) 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51ceffd640 1 -- 172.21.15.12:0/1442427598 learned_addr learned my addr 172.21.15.12:0/1442427598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51cffff640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d00d0690 0x7f51d01f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.704+0000 7f51cf7fe640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51d0072d80 0x7f51d0146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51ceffd640 1 -- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d00d0690 msgr2=0x7f51d01f30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51ceffd640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d00d0690 0x7f51d01f30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51ceffd640 1 -- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51d0072d80 msgr2=0x7f51d0146380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51ceffd640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51d0072d80 0x7f51d0146380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51ceffd640 1 -- 172.21.15.12:0/1442427598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d0143930 con 0x7f51d00af3e0 2024-04-15T15:23:19.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51cf7fe640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51d0072d80 0x7f51d0146380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:19.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51ceffd640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d01f2960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f51c402b930 tx=0x7f51c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:19.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51b7fff640 1 -- 172.21.15.12:0/1442427598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c403aa60 con 0x7f51d00af3e0 2024-04-15T15:23:19.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51b7fff640 1 -- 172.21.15.12:0/1442427598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c403ac20 con 0x7f51d00af3e0 2024-04-15T15:23:19.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51b7fff640 1 -- 172.21.15.12:0/1442427598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c40443f0 con 0x7f51d00af3e0 2024-04-15T15:23:19.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d01468c0 con 0x7f51d00af3e0 2024-04-15T15:23:19.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.705+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d0143bd0 con 0x7f51d00af3e0 2024-04-15T15:23:19.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.706+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d01f52e0 con 0x7f51d00af3e0 2024-04-15T15:23:19.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.707+0000 7f51b7fff640 1 -- 172.21.15.12:0/1442427598 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f51c404b060 con 0x7f51d00af3e0 2024-04-15T15:23:19.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.707+0000 7f51b7fff640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51bc06c980 0x7f51bc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:19.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.707+0000 7f51b7fff640 1 -- 172.21.15.12:0/1442427598 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f51c40adb00 con 0x7f51d00af3e0 2024-04-15T15:23:19.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.708+0000 7f51cf7fe640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51bc06c980 0x7f51bc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:19.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.708+0000 7f51cf7fe640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51bc06c980 0x7f51bc06ee40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f51c0006fd0 tx=0x7f51c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:19.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.712+0000 7f51b7fff640 1 -- 172.21.15.12:0/1442427598 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f51c403f070 con 0x7f51d00af3e0 2024-04-15T15:23:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.935+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51d00689e0 con 0x7f51d00af3e0 2024-04-15T15:23:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.936+0000 7f51b7fff640 1 -- 172.21.15.12:0/1442427598 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5180 (secure 0 0 0) 0x7f51c407bc30 con 0x7f51d00af3e0 2024-04-15T15:23:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:19.938 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5102,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/852515692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":852515692},{"type":"v1","addr":"172.21.15.55:6837","nonce":852515692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:23:19.042212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5200":{"gid":5200,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5200,"qdb_cluster":[5200]},"id":2}]} 2024-04-15T15:23:19.939 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-15T15:23:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51bc06c980 msgr2=0x7f51bc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51bc06c980 0x7f51bc06ee40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f51c0006fd0 tx=0x7f51c0008040 comp rx=0 tx=0).stop 2024-04-15T15:23:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 msgr2=0x7f51d01f2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d01f2960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f51c402b930 tx=0x7f51c40046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 shutdown_connections 2024-04-15T15:23:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51d00d0690 0x7f51d01f30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51d00af3e0 0x7f51d01f2960 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51bc06c980 0x7f51bc06ee40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 --2- 172.21.15.12:0/1442427598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51d0072d80 0x7f51d0146380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 >> 172.21.15.12:0/1442427598 conn(0x7f51d006be00 msgr2=0x7f51d00b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 shutdown_connections 2024-04-15T15:23:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:19.938+0000 7f51d5b24640 1 -- 172.21.15.12:0/1442427598 wait complete. 2024-04-15T15:23:20.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:22:32.656644+0000', 'modified': '2024-04-15T15:23:19.042212+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5200}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5200': {'gid': 5200, 'name': 'b', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.55:6836/1202155349', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 1202155349}, {'type': 'v1', 'addr': '172.21.15.55:6836', 'nonce': 1202155349}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5200, 'qdb_cluster': [5200]} 2024-04-15T15:23:20.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:23:20.354 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-15T15:23:20.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.461+0000 7f51cf50e640 1 Processor -- start 2024-04-15T15:23:20.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.461+0000 7f51cf50e640 1 -- start start 2024-04-15T15:23:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51cf50e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c815d090 0x7f51c8160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51cf50e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51c806f090 con 0x7f51c8067720 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51cf50e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51c815d660 con 0x7f51c806ec00 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51cf50e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f51c8160130 con 0x7f51c815d090 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51ccaa9640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f51c8067720 0x7f51c806e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47232/0 (socket says 172.21.15.12:47232) 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51ccaa9640 1 -- 172.21.15.12:0/4038181352 learned_addr learned my addr 172.21.15.12:0/4038181352 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c815d090 0x7f51c8160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461445152 0 0) 0x7f51c806f090 con 0x7f51c8067720 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.462+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51b0003520 con 0x7f51c8067720 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3114720600 0 0) 0x7f51b0003520 con 0x7f51c8067720 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c815d090 msgr2=0x7f51c8160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c815d090 0x7f51c8160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 >> v1:172.21.15.55:6789/0 conn(0x7f51c806ec00 legacy=0x7f51c80ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c815d940 con 0x7f51c8067720 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f51bc001670 con 0x7f51c8067720 2024-04-15T15:23:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 >> v1:172.21.15.12:6789/0 conn(0x7f51c8067720 legacy=0x7f51c806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51b00039e0 0x7f51b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c815d090 0x7f51b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51b0004d40 0x7f51b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51c8160130 con 0x7f51b0004d40 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51b000ba70 con 0x7f51b00039e0 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f51b000bc70 con 0x7f51c815d090 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51cdaab640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51b0004d40 0x7f51b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51b00039e0 0x7f51b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c815d090 0x7f51b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51cdaab640 1 -- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c815d090 msgr2=0x7f51b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51cdaab640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c815d090 0x7f51b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51cdaab640 1 -- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51b00039e0 msgr2=0x7f51b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51cdaab640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51b00039e0 0x7f51b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.463+0000 7f51cdaab640 1 -- 172.21.15.12:0/4038181352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b0003520 con 0x7f51b0004d40 2024-04-15T15:23:20.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.464+0000 7f51cdaab640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51b0004d40 0x7f51b00051c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f51b8002010 tx=0x7f51b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a0dd1cafa9928d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.464+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8039070 con 0x7f51b0004d40 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.464+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b802c750 con 0x7f51b0004d40 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.464+0000 7f51c7fff640 1 -- 172.21.15.12:0/4038181352 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8001be0 con 0x7f51b0004d40 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.465+0000 7f51cf50e640 1 -- 172.21.15.12:0/4038181352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51b0004d40 msgr2=0x7f51b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.465+0000 7f51cf50e640 1 --2- 172.21.15.12:0/4038181352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51b0004d40 0x7f51b00051c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f51b8002010 tx=0x7f51b0005830 comp rx=0 tx=0).stop 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.465+0000 7f51cdaab640 1 -- 172.21.15.12:0/4038181352 reap_dead start 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.465+0000 7f51cf50e640 1 -- 172.21.15.12:0/4038181352 shutdown_connections 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.465+0000 7f51cf50e640 1 -- 172.21.15.12:0/4038181352 >> 172.21.15.12:0/4038181352 conn(0x7f51c8073360 msgr2=0x7f51c806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.465+0000 7f51cf50e640 1 -- 172.21.15.12:0/4038181352 shutdown_connections 2024-04-15T15:23:20.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.465+0000 7f51cf50e640 1 -- 172.21.15.12:0/4038181352 wait complete. 2024-04-15T15:23:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.466+0000 7f51cf50e640 1 Processor -- start 2024-04-15T15:23:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.466+0000 7f51cf50e640 1 -- start start 2024-04-15T15:23:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cf50e640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c8148420 0x7f51c80abac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cf50e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51c80ac000 0x7f51c80a9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cf50e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c80a9cd0 0x7f51c80aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cf50e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51c80b0460 con 0x7f51c80ac000 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cf50e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51c8160660 con 0x7f51c80a9cd0 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cf50e640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f51c80b05e0 con 0x7f51c8148420 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51c80ac000 0x7f51c80a9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51c80ac000 0x7f51c80a9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35860/0 (socket says 172.21.15.12:35860) 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 -- 172.21.15.12:0/104634511 learned_addr learned my addr 172.21.15.12:0/104634511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cdaab640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c80a9cd0 0x7f51c80aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c8148420 0x7f51c80abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 -- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c8148420 msgr2=0x7f51c80abac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c8148420 0x7f51c80abac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 -- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c80a9cd0 msgr2=0x7f51c80aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c80a9cd0 0x7f51c80aa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 -- 172.21.15.12:0/104634511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c815a5b0 con 0x7f51c80ac000 2024-04-15T15:23:20.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51c80ac000 0x7f51c80a9790 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f51bc007290 tx=0x7f51bc007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51c5ffb640 1 -- 172.21.15.12:0/104634511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc00f940 con 0x7f51c80ac000 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51c5ffb640 1 -- 172.21.15.12:0/104634511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51bc018070 con 0x7f51c80ac000 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51cf50e640 1 -- 172.21.15.12:0/104634511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c815a890 con 0x7f51c80ac000 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.467+0000 7f51c5ffb640 1 -- 172.21.15.12:0/104634511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc00fd40 con 0x7f51c80ac000 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.468+0000 7f51cf50e640 1 -- 172.21.15.12:0/104634511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c815ae50 con 0x7f51c80ac000 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.469+0000 7f51c5ffb640 1 -- 172.21.15.12:0/104634511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f51bc01d460 con 0x7f51c80ac000 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.469+0000 7f51c5ffb640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51b406c980 0x7f51b406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.470+0000 7f51c5ffb640 1 -- 172.21.15.12:0/104634511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f51bc08be20 con 0x7f51c80ac000 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.470+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51b406c980 0x7f51b406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.470+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51b406c980 0x7f51b406ee40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f51c00099e0 tx=0x7f51c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.470+0000 7f51d21d3740 1 Processor -- start 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.470+0000 7f51d21d3740 1 -- start start 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 0x55b0a0f1f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0ed8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0ed9400 0x55b0a0f03ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51d21d3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b0a0e3e9c0 con 0x55b0a0f011d0 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51d21d3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b0a0de5650 con 0x55b0a0f21990 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51d21d3740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55b0a0e3a980 con 0x55b0a0ed9400 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51ccaa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0ed8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51ccaa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0ed8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35862/0 (socket says 172.21.15.12:35862) 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51ccaa9640 1 -- 172.21.15.12:0/1062763980 learned_addr learned my addr 172.21.15.12:0/1062763980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 0x55b0a0f1f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.471+0000 7f51cdaab640 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0ed9400 0x55b0a0f03ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51ccaa9640 1 -- 172.21.15.12:0/1062763980 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0ed9400 msgr2=0x55b0a0f03ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0ed9400 0x55b0a0f03ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51ccaa9640 1 -- 172.21.15.12:0/1062763980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 msgr2=0x55b0a0f1f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 0x55b0a0f1f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51ccaa9640 1 -- 172.21.15.12:0/1062763980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0a0b512d0 con 0x55b0a0f011d0 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0ed8ec0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f51bc08a110 tx=0x7f51bc007880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b73670dfd3f0ad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51aeffd640 1 -- 172.21.15.12:0/1062763980 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc059960 con 0x55b0a0f011d0 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51aeffd640 1 -- 172.21.15.12:0/1062763980 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51bc05dd10 con 0x55b0a0f011d0 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51aeffd640 1 -- 172.21.15.12:0/1062763980 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc062540 con 0x55b0a0f011d0 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51d21d3740 1 -- 172.21.15.12:0/1062763980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 msgr2=0x55b0a0ed8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.472+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0ed8ec0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f51bc08a110 tx=0x7f51bc007880 comp rx=0 tx=0).stop 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 -- 172.21.15.12:0/1062763980 shutdown_connections 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0ed9400 0x55b0a0f03ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0ed8ec0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1062763980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 0x55b0a0f1f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 -- 172.21.15.12:0/1062763980 >> 172.21.15.12:0/1062763980 conn(0x55b0a0ed6020 msgr2=0x55b0a0f1eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 -- 172.21.15.12:0/1062763980 shutdown_connections 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 -- 172.21.15.12:0/1062763980 wait complete. 2024-04-15T15:23:20.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 -- start start 2024-04-15T15:23:20.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.473+0000 7f51d21d3740 1 Processor -- start 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 0x55b0a0f36ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 0x55b0a0efc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51d21d3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b0a0ed8320 con 0x55b0a0f011d0 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51d21d3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b0a0df8a10 con 0x55b0a0f21990 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51d21d3740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55b0a0ed4650 con 0x55b0a0efbcd0 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51d21d3740 10 client.0 _ll_register_callbacks cb 0x55b0a0e34250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51cd2aa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51cd2aa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35878/0 (socket says 172.21.15.12:35878) 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51cd2aa640 1 -- 172.21.15.12:0/1604490780 learned_addr learned my addr 172.21.15.12:0/1604490780 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 0x55b0a0f36ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.474+0000 7f51cdaab640 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 0x55b0a0efc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51cd2aa640 1 -- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 msgr2=0x55b0a0efc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 0x55b0a0efc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51cd2aa640 1 -- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 msgr2=0x55b0a0f36ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 0x55b0a0f36ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51cd2aa640 1 -- 172.21.15.12:0/1604490780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0a0c1b6b0 con 0x55b0a0f011d0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1e1e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f51c0016710 tx=0x7f51c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7446c253d21e0a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51a7fff640 1 -- 172.21.15.12:0/1604490780 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0015600 con 0x55b0a0f011d0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51a7fff640 1 -- 172.21.15.12:0/1604490780 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f51c00157a0 con 0x55b0a0f011d0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 1 -- 172.21.15.12:0/1604490780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51bc05dd10 con 0x55b0a0f011d0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51a7fff640 1 -- 172.21.15.12:0/1604490780 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f51c0034050 con 0x55b0a0f011d0 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 10 client.6024 fetch_fsmap learned FSMap version 23 2024-04-15T15:23:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 1 -- 172.21.15.12:0/1604490780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55b0a0f371e0 con 0x55b0a0f011d0 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51a7fff640 1 -- 172.21.15.12:0/1604490780 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51bc05dd10 con 0x55b0a0f011d0 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51a7fff640 1 -- 172.21.15.12:0/1604490780 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f51c00159a0 con 0x55b0a0f011d0 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 10 client.6024 fetch_fsmap finished waiting for FSMap version 23 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 10 client.6024 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 20 client.6024 populate_metadata read hostname 'smithi012' 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] conn(0x55b0a0f1bb60 0x55b0a0f332b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 4 client.6024 mds_command: new command op to 5200 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.475+0000 7f51d21d3740 1 -- 172.21.15.12:0/1604490780 --> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0a0e4c3e0 con 0x55b0a0f1bb60 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.476+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] conn(0x55b0a0f1bb60 0x55b0a0f332b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.476+0000 7f51a7fff640 10 client.6024 ms_handle_connect on v2:172.21.15.55:6834/1202155349 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.476+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] conn(0x55b0a0f1bb60 0x55b0a0f332b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8024a18fc94570f7 server_cookie=88a907e77fea1724 in_seq=0 out_seq=0 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51a7fff640 1 -- 172.21.15.12:0/1604490780 <== mds.0 v2:172.21.15.55:6834/1202155349 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0a0e4c3e0 con 0x55b0a0f1bb60 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51a7fff640 10 client.6024 handle_command_reply: tid=0 2024-04-15T15:23:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 client.6024 shutdown 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 -- 172.21.15.12:0/1604490780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 msgr2=0x55b0a0f1e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1e1e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f51c0016710 tx=0x7f51c0004730 comp rx=0 tx=0).stop 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 -- 172.21.15.12:0/1604490780 shutdown_connections 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] conn(0x55b0a0f1bb60 0x55b0a0f332b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 0x55b0a0efc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51a7fff640 0 client.6024 ms_handle_reset on v2:172.21.15.55:6834/1202155349 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f21990 0x55b0a0f36ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 --2- 172.21.15.12:0/1604490780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1e1e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 -- 172.21.15.12:0/1604490780 >> 172.21.15.12:0/1604490780 conn(0x55b0a0ed9400 msgr2=0x55b0a0ed2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 -- 172.21.15.12:0/1604490780 shutdown_connections 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 1 -- 172.21.15.12:0/1604490780 wait complete. 2024-04-15T15:23:20.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.477+0000 7f51d21d3740 20 client.6024 trim_cache size 0 max 16384 2024-04-15T15:23:20.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.495+0000 7f51d21d3740 1 Processor -- start 2024-04-15T15:23:20.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.495+0000 7f51d21d3740 1 -- start start 2024-04-15T15:23:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 0x55b0a0f17680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f21990 0x55b0a0f1ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51d21d3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51c81fda80 con 0x55b0a0f21990 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51d21d3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51c81fcb20 con 0x55b0a0f011d0 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51d21d3740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f51c80b5bf0 con 0x55b0a0efbcd0 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cdaab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f21990 0x55b0a0f1ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cdaab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f21990 0x55b0a0f1ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35884/0 (socket says 172.21.15.12:35884) 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cdaab640 1 -- 172.21.15.12:0/2630252913 learned_addr learned my addr 172.21.15.12:0/2630252913 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 0x55b0a0f17680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cdaab640 1 -- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 msgr2=0x55b0a0f17680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cdaab640 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 0x55b0a0f17680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cdaab640 1 -- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f011d0 msgr2=0x55b0a0f1f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cdaab640 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.496+0000 7f51cdaab640 1 -- 172.21.15.12:0/2630252913 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0a0efc110 con 0x55b0a0f21990 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:20.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51cdaab640 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f21990 0x55b0a0f1ebf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f51b802c1c0 tx=0x7f51b802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8417c57dd8f0a1d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51ae7fc640 1 -- 172.21.15.12:0/2630252913 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8039040 con 0x55b0a0f21990 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51ae7fc640 1 -- 172.21.15.12:0/2630252913 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b8006b50 con 0x55b0a0f21990 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51ae7fc640 1 -- 172.21.15.12:0/2630252913 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8003970 con 0x55b0a0f21990 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 -- 172.21.15.12:0/2630252913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f21990 msgr2=0x55b0a0f1ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f21990 0x55b0a0f1ebf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f51b802c1c0 tx=0x7f51b802e000 comp rx=0 tx=0).stop 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 -- 172.21.15.12:0/2630252913 shutdown_connections 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f21990 0x55b0a0f1ebf0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0f011d0 0x55b0a0f1f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 --2- 172.21.15.12:0/2630252913 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0efbcd0 0x55b0a0f17680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 -- 172.21.15.12:0/2630252913 >> 172.21.15.12:0/2630252913 conn(0x55b0a0ed9400 msgr2=0x55b0a0ecf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 -- 172.21.15.12:0/2630252913 shutdown_connections 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.497+0000 7f51d21d3740 1 -- 172.21.15.12:0/2630252913 wait complete. 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.498+0000 7f51d21d3740 1 -- start start 2024-04-15T15:23:20.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.498+0000 7f51d21d3740 1 Processor -- start 2024-04-15T15:23:20.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0efbcd0 0x55b0a0f172d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f9df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51d21d3740 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0f21990 0x55b0a0f9e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51d21d3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b0a0f190f0 con 0x55b0a0f011d0 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51d21d3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51c80af1d0 con 0x55b0a0efbcd0 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51d21d3740 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x55b0a0efc690 con 0x55b0a0f21990 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51d21d3740 10 client.0 _ll_register_callbacks cb 0x7f51c0013a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f9df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f9df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35890/0 (socket says 172.21.15.12:35890) 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 -- 172.21.15.12:0/631104778 learned_addr learned my addr 172.21.15.12:0/631104778 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0efbcd0 0x55b0a0f172d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51cdaab640 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0f21990 0x55b0a0f9e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 -- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0f21990 msgr2=0x55b0a0f9e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0f21990 0x55b0a0f9e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 -- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0efbcd0 msgr2=0x55b0a0f172d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0efbcd0 0x55b0a0f172d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 -- 172.21.15.12:0/631104778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0a0f06020 con 0x55b0a0f011d0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51cdaab640 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0f21990 0x55b0a0f9e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51ccaa9640 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f9df90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f51bc062010 tx=0x7f51bc015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8362eff79d2401 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.499+0000 7f51a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51a7fff640 1 -- 172.21.15.12:0/631104778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc055c00 con 0x55b0a0f011d0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51a7fff640 1 -- 172.21.15.12:0/631104778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f51bc0600c0 con 0x55b0a0f011d0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 1 -- 172.21.15.12:0/631104778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51b8006b50 con 0x55b0a0f011d0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51a7fff640 1 -- 172.21.15.12:0/631104778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f51bc059960 con 0x55b0a0f011d0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 10 client.6030 fetch_fsmap learned FSMap version 23 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 1 -- 172.21.15.12:0/631104778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55b0a0f16ad0 con 0x55b0a0f011d0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51a7fff640 1 -- 172.21.15.12:0/631104778 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b8006b50 con 0x55b0a0f011d0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51a7fff640 1 -- 172.21.15.12:0/631104778 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f51bc058db0 con 0x55b0a0f011d0 2024-04-15T15:23:20.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 10 client.6030 fetch_fsmap finished waiting for FSMap version 23 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 10 client.6030 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 20 client.6030 populate_metadata read hostname 'smithi012' 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] conn(0x55b0a0f361e0 0x55b0a0f46d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 4 client.6030 mds_command: new command op to 5200 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51d21d3740 1 -- 172.21.15.12:0/631104778 --> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] -- command(tid 0: {"prefix": "status"}) -- 0x55b0a0e46250 con 0x55b0a0f361e0 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.500+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] conn(0x55b0a0f361e0 0x55b0a0f46d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51a7fff640 10 client.6030 ms_handle_connect on v2:172.21.15.55:6834/1202155349 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51cd2aa640 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] conn(0x55b0a0f361e0 0x55b0a0f46d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfb230fd389705aa server_cookie=58d97a5578787ef3 in_seq=0 out_seq=0 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51a7fff640 1 -- 172.21.15.12:0/631104778 <== mds.0 v2:172.21.15.55:6834/1202155349 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55b0a0e46250 con 0x55b0a0f361e0 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51a7fff640 10 client.6030 handle_command_reply: tid=0 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 client.6030 shutdown 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 -- 172.21.15.12:0/631104778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 msgr2=0x55b0a0f9df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f9df90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f51bc062010 tx=0x7f51bc015070 comp rx=0 tx=0).stop 2024-04-15T15:23:20.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 -- 172.21.15.12:0/631104778 shutdown_connections 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:6834/1202155349,v1:172.21.15.55:6836/1202155349] conn(0x55b0a0f361e0 0x55b0a0f46d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x55b0a0f21990 0x55b0a0f9e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b0a0f011d0 0x55b0a0f9df90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 --2- 172.21.15.12:0/631104778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0a0efbcd0 0x55b0a0f172d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51a7fff640 0 client.6030 ms_handle_reset on v2:172.21.15.55:6834/1202155349 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 -- 172.21.15.12:0/631104778 >> 172.21.15.12:0/631104778 conn(0x55b0a0ed9400 msgr2=0x55b0a0f47f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 -- 172.21.15.12:0/631104778 shutdown_connections 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 1 -- 172.21.15.12:0/631104778 wait complete. 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.501+0000 7f51d21d3740 20 client.6030 trim_cache size 0 max 16384 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "928fa63b-2322-46eb-a6b8-ef2b880cd7c9", 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 5200, 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 4.4811526410000004, 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 23, 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 42, 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 42, 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 54.307343420999999 2024-04-15T15:23:20.511 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 -- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51b406c980 msgr2=0x7f51b406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51b406c980 0x7f51b406ee40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f51c00099e0 tx=0x7f51c0008040 comp rx=0 tx=0).stop 2024-04-15T15:23:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 -- 172.21.15.12:0/104634511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51c80ac000 msgr2=0x7f51c80a9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51c80ac000 0x7f51c80a9790 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f51bc007290 tx=0x7f51bc007750 comp rx=0 tx=0).stop 2024-04-15T15:23:20.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 -- 172.21.15.12:0/104634511 shutdown_connections 2024-04-15T15:23:20.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f51b406c980 0x7f51b406ee40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c80a9cd0 0x7f51c80aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51c80ac000 0x7f51c80a9790 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 --2- 172.21.15.12:0/104634511 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f51c8148420 0x7f51c80abac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 -- 172.21.15.12:0/104634511 >> 172.21.15.12:0/104634511 conn(0x7f51c8073360 msgr2=0x7f51c806c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:20.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 -- 172.21.15.12:0/104634511 shutdown_connections 2024-04-15T15:23:20.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.502+0000 7f51cf50e640 1 -- 172.21.15.12:0/104634511 wait complete. 2024-04-15T15:23:20.515 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-15T15:23:20.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:23:20.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:23:20.617 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-15T15:23:20.617 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:23:20.617 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:23:20.618 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:20.633 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:23:20.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.775+0000 7f0f91433640 1 Processor -- start 2024-04-15T15:23:20.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.775+0000 7f0f91433640 1 -- start start 2024-04-15T15:23:20.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.776+0000 7f0f91433640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f8c15ebb0 0x7f0f8c06cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.776+0000 7f0f91433640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f8c0ced60 con 0x7f0f8c09a9d0 2024-04-15T15:23:20.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.776+0000 7f0f91433640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f8c0cccb0 con 0x7f0f8c09b780 2024-04-15T15:23:20.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.776+0000 7f0f91433640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0f8c0cd8c0 con 0x7f0f8c15ebb0 2024-04-15T15:23:20.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.776+0000 7f0f8affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0f8c09a9d0 0x7f0f8c09adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47238/0 (socket says 172.21.15.12:47238) 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.776+0000 7f0f8affd640 1 -- 172.21.15.12:0/240847582 learned_addr learned my addr 172.21.15.12:0/240847582 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.776+0000 7f0f8a7fc640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f8c15ebb0 0x7f0f8c06cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2966613461 0 0) 0x7f0f7c001490 con 0x7f0f8c09a9d0 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f78003520 con 0x7f0f8c09a9d0 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1870982628 0 0) 0x7f0f78003520 con 0x7f0f8c09a9d0 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f8c15ebb0 msgr2=0x7f0f8c06cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f8c15ebb0 0x7f0f8c06cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 >> v1:172.21.15.55:6789/0 conn(0x7f0f8c09b780 legacy=0x7f0f8c0b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8c0ccf90 con 0x7f0f8c09a9d0 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f0f7c001bc0 con 0x7f0f8c09a9d0 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 >> v1:172.21.15.12:6789/0 conn(0x7f0f8c09a9d0 legacy=0x7f0f8c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780039e0 0x7f0f78003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f78004800 0x7f0f78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780051c0 0x7f0f7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c0cd8c0 con 0x7f0f780039e0 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7800bf00 con 0x7f0f78004800 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0f7800c080 con 0x7f0f780051c0 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8affd640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780039e0 0x7f0f78003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8a7fc640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f78004800 0x7f0f78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8b7fe640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780051c0 0x7f0f7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8affd640 1 -- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780051c0 msgr2=0x7f0f7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8affd640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780051c0 0x7f0f7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8affd640 1 -- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f78004800 msgr2=0x7f0f78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8affd640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f78004800 0x7f0f78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8affd640 1 -- 172.21.15.12:0/240847582 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f78003520 con 0x7f0f780039e0 2024-04-15T15:23:20.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.777+0000 7f0f8a7fc640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f78004800 0x7f0f78004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.778+0000 7f0f8affd640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780039e0 0x7f0f78003e00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00aa50 tx=0x7f0f78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c9f841946d9715 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.778+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c038070 con 0x7f0f780039e0 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.778+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f7c033650 con 0x7f0f780039e0 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.778+0000 7f0f89ffb640 1 -- 172.21.15.12:0/240847582 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00a670 con 0x7f0f780039e0 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.779+0000 7f0f91433640 1 -- 172.21.15.12:0/240847582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780039e0 msgr2=0x7f0f78003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.779+0000 7f0f91433640 1 --2- 172.21.15.12:0/240847582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780039e0 0x7f0f78003e00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00aa50 tx=0x7f0f78005830 comp rx=0 tx=0).stop 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.779+0000 7f0f8b7fe640 1 -- 172.21.15.12:0/240847582 reap_dead start 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.779+0000 7f0f91433640 1 -- 172.21.15.12:0/240847582 shutdown_connections 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.779+0000 7f0f91433640 1 -- 172.21.15.12:0/240847582 >> 172.21.15.12:0/240847582 conn(0x7f0f8c0b3fb0 msgr2=0x7f0f8c0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.779+0000 7f0f91433640 1 -- 172.21.15.12:0/240847582 shutdown_connections 2024-04-15T15:23:20.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.779+0000 7f0f91433640 1 -- 172.21.15.12:0/240847582 wait complete. 2024-04-15T15:23:20.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.780+0000 7f0f91433640 1 Processor -- start 2024-04-15T15:23:20.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.780+0000 7f0f91433640 1 -- start start 2024-04-15T15:23:20.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f91433640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780039e0 0x7f0f8c151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f91433640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c0c5ff0 0x7f0f8c0c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f91433640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0c6450 0x7f0f8c0c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f91433640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c0b5310 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f91433640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c0cd550 con 0x7f0f8c0c6450 2024-04-15T15:23:20.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f91433640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0f8c0b5490 con 0x7f0f780039e0 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c0c5ff0 0x7f0f8c0c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c0c5ff0 0x7f0f8c0c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35894/0 (socket says 172.21.15.12:35894) 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8a7fc640 1 -- 172.21.15.12:0/2616773634 learned_addr learned my addr 172.21.15.12:0/2616773634 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8affd640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780039e0 0x7f0f8c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8b7fe640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0c6450 0x7f0f8c0c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8a7fc640 1 -- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780039e0 msgr2=0x7f0f8c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8a7fc640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780039e0 0x7f0f8c151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8a7fc640 1 -- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0c6450 msgr2=0x7f0f8c0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8a7fc640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0c6450 0x7f0f8c0c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8a7fc640 1 -- 172.21.15.12:0/2616773634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8c0c7a50 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.781+0000 7f0f8b7fe640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0c6450 0x7f0f8c0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.782+0000 7f0f8a7fc640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c0c5ff0 0x7f0f8c0c8120 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0f74011660 tx=0x7f0f74011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.782+0000 7f0f837fe640 1 -- 172.21.15.12:0/2616773634 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7400f9c0 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.782+0000 7f0f837fe640 1 -- 172.21.15.12:0/2616773634 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f7400fb60 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.782+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f8c0c69b0 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.782+0000 7f0f837fe640 1 -- 172.21.15.12:0/2616773634 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74015460 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.782+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f8c0c6f70 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.783+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f8c0690f0 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.783+0000 7f0f837fe640 1 -- 172.21.15.12:0/2616773634 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0f74005990 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.784+0000 7f0f837fe640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f6c06c9b0 0x7f0f6c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.784+0000 7f0f837fe640 1 -- 172.21.15.12:0/2616773634 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0f7408c570 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.784+0000 7f0f8affd640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f6c06c9b0 0x7f0f6c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:20.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.784+0000 7f0f8affd640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f6c06c9b0 0x7f0f6c06ee70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00aa20 tx=0x7f0f7c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:20.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.788+0000 7f0f837fe640 1 -- 172.21.15.12:0/2616773634 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f74019030 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.978+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f8c15df70 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.979+0000 7f0f837fe640 1 -- 172.21.15.12:0/2616773634 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17084 (secure 0 0 0) 0x7f0f7405a7b0 con 0x7f0f8c0c5ff0 2024-04-15T15:23:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:20.982 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":42,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:23:15.001862+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6837/3388147599":"2024-04-16T15:23:15.001839+0000","172.21.15.55:6835/3388147599":"2024-04-16T15:23:15.001839+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:23:20.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.981+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f6c06c9b0 msgr2=0x7f0f6c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.981+0000 7f0f91433640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f6c06c9b0 0x7f0f6c06ee70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00aa20 tx=0x7f0f7c030000 comp rx=0 tx=0).stop 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.981+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c0c5ff0 msgr2=0x7f0f8c0c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.981+0000 7f0f91433640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c0c5ff0 0x7f0f8c0c8120 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0f74011660 tx=0x7f0f74011b20 comp rx=0 tx=0).stop 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.982+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 shutdown_connections 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.982+0000 7f0f91433640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f6c06c9b0 0x7f0f6c06ee70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.982+0000 7f0f91433640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f8c0c6450 0x7f0f8c0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.982+0000 7f0f91433640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f8c0c5ff0 0x7f0f8c0c8120 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.982+0000 7f0f91433640 1 --2- 172.21.15.12:0/2616773634 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f780039e0 0x7f0f8c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.982+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 >> 172.21.15.12:0/2616773634 conn(0x7f0f8c0b3fb0 msgr2=0x7f0f8c15d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.982+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 shutdown_connections 2024-04-15T15:23:20.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:20.982+0000 7f0f91433640 1 -- 172.21.15.12:0/2616773634 wait complete. 2024-04-15T15:23:20.995 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:23:20.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:23:21.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.102+0000 7fe962be2640 1 Processor -- start 2024-04-15T15:23:21.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.102+0000 7fe962be2640 1 -- start start 2024-04-15T15:23:21.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.102+0000 7fe962be2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe95c0b5230 0x7fe95c072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.102+0000 7fe962be2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe95c072c60 0x7fe95c06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe962be2640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe95c073080 con 0x7fe95c158770 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe962be2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe95c0b4da0 con 0x7fe95c0b5230 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe962be2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe95c0b4190 con 0x7fe95c072c60 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe95bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe95c0b5230 0x7fe95c072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe95bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe95c0b5230 0x7fe95c072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35902/0 (socket says 172.21.15.12:35902) 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe95bfff640 1 -- 172.21.15.12:0/2686361650 learned_addr learned my addr 172.21.15.12:0/2686361650 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe95bfff640 1 -- 172.21.15.12:0/2686361650 >> v2:172.21.15.55:3300/0 conn(0x7fe95c072c60 msgr2=0x7fe95c06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe95bfff640 1 --2- 172.21.15.12:0/2686361650 >> v2:172.21.15.55:3300/0 conn(0x7fe95c072c60 0x7fe95c06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe95bfff640 1 -- 172.21.15.12:0/2686361650 >> v1:172.21.15.55:6789/0 conn(0x7fe95c158770 legacy=0x7fe95c15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:21.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.103+0000 7fe95bfff640 1 -- 172.21.15.12:0/2686361650 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95c0b4470 con 0x7fe95c0b5230 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe95bfff640 1 --2- 172.21.15.12:0/2686361650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c0b5230 0x7fe95c072720 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe95c072640 tx=0x7fe94c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4999245ca40e0ad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe95b7fe640 1 -- 172.21.15.12:0/2686361650 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c035020 con 0x7fe95c0b5230 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe95b7fe640 1 -- 172.21.15.12:0/2686361650 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe94c031870 con 0x7fe95c0b5230 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe95b7fe640 1 -- 172.21.15.12:0/2686361650 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c031c40 con 0x7fe95c0b5230 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe962be2640 1 -- 172.21.15.12:0/2686361650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c0b5230 msgr2=0x7fe95c072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe962be2640 1 --2- 172.21.15.12:0/2686361650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c0b5230 0x7fe95c072720 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe95c072640 tx=0x7fe94c02b910 comp rx=0 tx=0).stop 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe962be2640 1 -- 172.21.15.12:0/2686361650 shutdown_connections 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe962be2640 1 --2- 172.21.15.12:0/2686361650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c0b5230 0x7fe95c072720 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe962be2640 1 --2- 172.21.15.12:0/2686361650 >> v2:172.21.15.55:3300/0 conn(0x7fe95c072c60 0x7fe95c06dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.104+0000 7fe962be2640 1 -- 172.21.15.12:0/2686361650 >> 172.21.15.12:0/2686361650 conn(0x7fe95c06be00 msgr2=0x7fe95c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:21.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.105+0000 7fe962be2640 1 -- 172.21.15.12:0/2686361650 shutdown_connections 2024-04-15T15:23:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.105+0000 7fe962be2640 1 -- 172.21.15.12:0/2686361650 wait complete. 2024-04-15T15:23:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.106+0000 7fe962be2640 1 Processor -- start 2024-04-15T15:23:21.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.106+0000 7fe962be2640 1 -- start start 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe962be2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c072c60 0x7fe95c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe962be2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe95c0b5230 0x7fe95c1e6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe962be2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe95c158770 0x7fe95c15d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe962be2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c071fa0 con 0x7fe95c072c60 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe962be2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c0b4a30 con 0x7fe95c0b5230 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe962be2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe95c158b50 con 0x7fe95c158770 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c072c60 0x7fe95c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c072c60 0x7fe95c15b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35908/0 (socket says 172.21.15.12:35908) 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 -- 172.21.15.12:0/777807682 learned_addr learned my addr 172.21.15.12:0/777807682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:21.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe95bfff640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe95c0b5230 0x7fe95c1e6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96117f640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe95c158770 0x7fe95c15d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 -- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe95c158770 msgr2=0x7fe95c15d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe95c158770 0x7fe95c15d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 -- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe95c0b5230 msgr2=0x7fe95c1e6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe95c0b5230 0x7fe95c1e6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 -- 172.21.15.12:0/777807682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95c15b780 con 0x7fe95c072c60 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe96097e640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c072c60 0x7fe95c15b240 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe95400ba50 tx=0x7fe95400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe9597fa640 1 -- 172.21.15.12:0/777807682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954019070 con 0x7fe95c072c60 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe9597fa640 1 -- 172.21.15.12:0/777807682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe95400cc10 con 0x7fe95c072c60 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95c151720 con 0x7fe95c072c60 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.107+0000 7fe9597fa640 1 -- 172.21.15.12:0/777807682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954014440 con 0x7fe95c072c60 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.108+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95c15bce0 con 0x7fe95c072c60 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.109+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe95c1518f0 con 0x7fe95c072c60 2024-04-15T15:23:21.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.109+0000 7fe9597fa640 1 -- 172.21.15.12:0/777807682 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe95401c070 con 0x7fe95c072c60 2024-04-15T15:23:21.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.109+0000 7fe9597fa640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe93c06c980 0x7fe93c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.109+0000 7fe9597fa640 1 -- 172.21.15.12:0/777807682 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fe95408bb30 con 0x7fe95c072c60 2024-04-15T15:23:21.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.109+0000 7fe95bfff640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe93c06c980 0x7fe93c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.110+0000 7fe95bfff640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe93c06c980 0x7fe93c06ee40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe94c036000 tx=0x7fe94c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.114+0000 7fe9597fa640 1 -- 172.21.15.12:0/777807682 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe95c15b780 con 0x7fe95c072c60 2024-04-15T15:23:21.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.300+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe95c0689e0 con 0x7fe95c072c60 2024-04-15T15:23:21.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.300+0000 7fe9597fa640 1 -- 172.21.15.12:0/777807682 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fe954059e00 con 0x7fe95c072c60 2024-04-15T15:23:21.303 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe93c06c980 msgr2=0x7fe93c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe93c06c980 0x7fe93c06ee40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe94c036000 tx=0x7fe94c03c040 comp rx=0 tx=0).stop 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c072c60 msgr2=0x7fe95c15b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c072c60 0x7fe95c15b240 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe95400ba50 tx=0x7fe95400bf10 comp rx=0 tx=0).stop 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 shutdown_connections 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe95c158770 0x7fe95c15d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe93c06c980 0x7fe93c06ee40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe95c0b5230 0x7fe95c1e6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.302+0000 7fe962be2640 1 --2- 172.21.15.12:0/777807682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe95c072c60 0x7fe95c15b240 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.303+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 >> 172.21.15.12:0/777807682 conn(0x7fe95c06be00 msgr2=0x7fe95c0b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:21.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.303+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 shutdown_connections 2024-04-15T15:23:21.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.303+0000 7fe962be2640 1 -- 172.21.15.12:0/777807682 wait complete. 2024-04-15T15:23:21.317 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:23:21.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.423+0000 7f6a9e332640 1 Processor -- start 2024-04-15T15:23:21.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.423+0000 7f6a9e332640 1 -- start start 2024-04-15T15:23:21.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.423+0000 7f6a9e332640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6a980d2360 0x7f6a980d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.423+0000 7f6a9e332640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.423+0000 7f6a9e332640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a980b47a0 con 0x7f6a9815e700 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.423+0000 7f6a9e332640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6a980b53b0 con 0x7f6a980d2360 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.423+0000 7f6a9e332640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6a980b4a80 con 0x7f6a9815eae0 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a9c8cf640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a977fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6a980d2360 0x7f6a980d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a97fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6a9815e700 0x7f6a980b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:54598/0 (socket says 172.21.15.12:54598) 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a977fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6a980d2360 0x7f6a980d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:50464/0 (socket says 172.21.15.12:50464) 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a977fe640 1 -- 172.21.15.12:0/4159754092 learned_addr learned my addr 172.21.15.12:0/4159754092 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a9c8cf640 1 -- 172.21.15.12:0/4159754092 >> v2:172.21.15.55:3300/0 conn(0x7f6a980d2360 msgr2=0x7f6a980d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a9c8cf640 1 --2- 172.21.15.12:0/4159754092 >> v2:172.21.15.55:3300/0 conn(0x7f6a980d2360 0x7f6a980d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a9c8cf640 1 -- 172.21.15.12:0/4159754092 >> v1:172.21.15.55:6789/0 conn(0x7f6a9815e700 legacy=0x7f6a980b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a9c8cf640 1 -- 172.21.15.12:0/4159754092 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a980b4d60 con 0x7f6a9815eae0 2024-04-15T15:23:21.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.424+0000 7f6a977fe640 1 --2- 172.21.15.12:0/4159754092 >> v2:172.21.15.55:3300/0 conn(0x7f6a980d2360 0x7f6a980d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.425+0000 7f6a9c8cf640 1 --2- 172.21.15.12:0/4159754092 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980d30a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009d90 tx=0x7f6a9002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc1784dc87cc9f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.425+0000 7f6a96ffd640 1 -- 172.21.15.12:0/4159754092 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90039070 con 0x7f6a9815eae0 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.425+0000 7f6a96ffd640 1 -- 172.21.15.12:0/4159754092 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a9002cc50 con 0x7f6a9815eae0 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.425+0000 7f6a96ffd640 1 -- 172.21.15.12:0/4159754092 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90034440 con 0x7f6a9815eae0 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.426+0000 7f6a9e332640 1 -- 172.21.15.12:0/4159754092 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 msgr2=0x7f6a980d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.426+0000 7f6a9e332640 1 --2- 172.21.15.12:0/4159754092 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980d30a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009d90 tx=0x7f6a9002bea0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.426+0000 7f6a9e332640 1 -- 172.21.15.12:0/4159754092 shutdown_connections 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.426+0000 7f6a9e332640 1 --2- 172.21.15.12:0/4159754092 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980d30a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.426+0000 7f6a9e332640 1 --2- 172.21.15.12:0/4159754092 >> v2:172.21.15.55:3300/0 conn(0x7f6a980d2360 0x7f6a980d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.426+0000 7f6a9e332640 1 -- 172.21.15.12:0/4159754092 >> 172.21.15.12:0/4159754092 conn(0x7f6a9808bf60 msgr2=0x7f6a980d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:21.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.426+0000 7f6a9e332640 1 -- 172.21.15.12:0/4159754092 shutdown_connections 2024-04-15T15:23:21.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.426+0000 7f6a9e332640 1 -- 172.21.15.12:0/4159754092 wait complete. 2024-04-15T15:23:21.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.427+0000 7f6a9e332640 1 Processor -- start 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.427+0000 7f6a9e332640 1 -- start start 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a9e332640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a980d2360 0x7f6a980ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a9e332640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a9815e700 0x7f6a980b9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a9e332640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a9e332640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6a980d1d40 con 0x7f6a9815e700 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a9e332640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6a980b4450 con 0x7f6a980d2360 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a9e332640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6a980d1ec0 con 0x7f6a9815eae0 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a9815e700 0x7f6a980b9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a9815e700 0x7f6a980b9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35922/0 (socket says 172.21.15.12:35922) 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a977fe640 1 -- 172.21.15.12:0/1869393167 learned_addr learned my addr 172.21.15.12:0/1869393167 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a97fff640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a980d2360 0x7f6a980ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.428+0000 7f6a9c8cf640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a977fe640 1 -- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 msgr2=0x7f6a980b9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a977fe640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980b9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a977fe640 1 -- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a980d2360 msgr2=0x7f6a980ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a977fe640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a980d2360 0x7f6a980ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a977fe640 1 -- 172.21.15.12:0/1869393167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9815f940 con 0x7f6a9815e700 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a977fe640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a9815e700 0x7f6a980b9530 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00e9d0 tx=0x7f6a8c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a94ff9640 1 -- 172.21.15.12:0/1869393167 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c017070 con 0x7f6a9815e700 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a94ff9640 1 -- 172.21.15.12:0/1869393167 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a8c00ca60 con 0x7f6a9815e700 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9815fc20 con 0x7f6a9815e700 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a94ff9640 1 -- 172.21.15.12:0/1869393167 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c00ce10 con 0x7f6a9815e700 2024-04-15T15:23:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.429+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a981601e0 con 0x7f6a9815e700 2024-04-15T15:23:21.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.430+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a980689e0 con 0x7f6a9815e700 2024-04-15T15:23:21.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.431+0000 7f6a94ff9640 1 -- 172.21.15.12:0/1869393167 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6a8c01a070 con 0x7f6a9815e700 2024-04-15T15:23:21.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.431+0000 7f6a94ff9640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6a8406c980 0x7f6a8406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.431+0000 7f6a94ff9640 1 -- 172.21.15.12:0/1869393167 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f6a8c08c080 con 0x7f6a9815e700 2024-04-15T15:23:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.431+0000 7f6a97fff640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6a8406c980 0x7f6a8406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.432+0000 7f6a97fff640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6a8406c980 0x7f6a8406ee40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6a88000fb0 tx=0x7f6a88007e70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.435+0000 7f6a94ff9640 1 -- 172.21.15.12:0/1869393167 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6a8c059bf0 con 0x7f6a9815e700 2024-04-15T15:23:21.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.620+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6a980ba500 con 0x7f6a9815e700 2024-04-15T15:23:21.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.620+0000 7f6a94ff9640 1 -- 172.21.15.12:0/1869393167 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f6a8c05def0 con 0x7f6a9815e700 2024-04-15T15:23:21.623 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:23:21.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6a8406c980 msgr2=0x7f6a8406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6a8406c980 0x7f6a8406ee40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6a88000fb0 tx=0x7f6a88007e70 comp rx=0 tx=0).stop 2024-04-15T15:23:21.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a9815e700 msgr2=0x7f6a980b9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a9815e700 0x7f6a980b9530 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00e9d0 tx=0x7f6a8c00ee90 comp rx=0 tx=0).stop 2024-04-15T15:23:21.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 shutdown_connections 2024-04-15T15:23:21.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6a9815eae0 0x7f6a980b9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a9815e700 0x7f6a980b9530 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6a8406c980 0x7f6a8406ee40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 --2- 172.21.15.12:0/1869393167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a980d2360 0x7f6a980ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.622+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 >> 172.21.15.12:0/1869393167 conn(0x7f6a9808bf60 msgr2=0x7f6a980d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.623+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 shutdown_connections 2024-04-15T15:23:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.623+0000 7f6a9e332640 1 -- 172.21.15.12:0/1869393167 wait complete. 2024-04-15T15:23:21.636 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:23:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.740+0000 7f160dfe6640 1 Processor -- start 2024-04-15T15:23:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.740+0000 7f160dfe6640 1 -- start start 2024-04-15T15:23:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f160dfe6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f16080d2360 0x7f16080d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f160dfe6640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f160815eae0 0x7f16080d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f160dfe6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16080b47a0 con 0x7f160815e700 2024-04-15T15:23:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f160dfe6640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f16080b53b0 con 0x7f16080d2360 2024-04-15T15:23:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f160dfe6640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f16080b4a80 con 0x7f160815eae0 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f16077fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f160815e700 0x7f16080b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47242/0 (socket says 172.21.15.12:47242) 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f16077fe640 1 -- 172.21.15.12:0/624425743 learned_addr learned my addr 172.21.15.12:0/624425743 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f1606ffd640 1 --2- 172.21.15.12:0/624425743 >> v2:172.21.15.55:3300/0 conn(0x7f16080d2360 0x7f16080d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.741+0000 7f1607fff640 1 --2- 172.21.15.12:0/624425743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f160815eae0 0x7f16080d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.742+0000 7f16067fc640 1 -- 172.21.15.12:0/624425743 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278978324 0 0) 0x7f16080b47a0 con 0x7f160815e700 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.742+0000 7f16067fc640 1 -- 172.21.15.12:0/624425743 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15f0003520 con 0x7f160815e700 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.742+0000 7f1606ffd640 1 -- 172.21.15.12:0/624425743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f160815eae0 msgr2=0x7f16080d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.742+0000 7f1606ffd640 1 --2- 172.21.15.12:0/624425743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f160815eae0 0x7f16080d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.742+0000 7f1606ffd640 1 -- 172.21.15.12:0/624425743 >> v1:172.21.15.12:6789/0 conn(0x7f160815e700 legacy=0x7f16080b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.742+0000 7f16067fc640 1 -- 172.21.15.12:0/624425743 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3807286889 0 0) 0x7f15f0003520 con 0x7f160815e700 2024-04-15T15:23:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.742+0000 7f1606ffd640 1 -- 172.21.15.12:0/624425743 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16080b4d60 con 0x7f16080d2360 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f1606ffd640 1 --2- 172.21.15.12:0/624425743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16080d2360 0x7f16080d2b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0097a0 tx=0x7f15fc02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d2507c87fb191a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f16067fc640 1 -- 172.21.15.12:0/624425743 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc037070 con 0x7f16080d2360 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f16067fc640 1 -- 172.21.15.12:0/624425743 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15fc030510 con 0x7f16080d2360 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f16067fc640 1 -- 172.21.15.12:0/624425743 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8001810 con 0x7f16080d2360 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f160dfe6640 1 -- 172.21.15.12:0/624425743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16080d2360 msgr2=0x7f16080d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f160dfe6640 1 --2- 172.21.15.12:0/624425743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16080d2360 0x7f16080d2b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0097a0 tx=0x7f15fc02b790 comp rx=0 tx=0).stop 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f160dfe6640 1 -- 172.21.15.12:0/624425743 shutdown_connections 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f160dfe6640 1 --2- 172.21.15.12:0/624425743 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f160815eae0 0x7f16080d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f160dfe6640 1 --2- 172.21.15.12:0/624425743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16080d2360 0x7f16080d2b60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f160dfe6640 1 -- 172.21.15.12:0/624425743 >> 172.21.15.12:0/624425743 conn(0x7f160808bf60 msgr2=0x7f16080d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f160dfe6640 1 -- 172.21.15.12:0/624425743 shutdown_connections 2024-04-15T15:23:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.743+0000 7f160dfe6640 1 -- 172.21.15.12:0/624425743 wait complete. 2024-04-15T15:23:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.745+0000 7f160dfe6640 1 Processor -- start 2024-04-15T15:23:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.745+0000 7f160dfe6640 1 -- start start 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.745+0000 7f160dfe6640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f16080d2360 0x7f1608161a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.745+0000 7f160dfe6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160815e700 0x7f16081f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.745+0000 7f160dfe6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f160815eae0 0x7f16081f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.745+0000 7f160dfe6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f16080d1d40 con 0x7f160815e700 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.745+0000 7f160dfe6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f16080b4450 con 0x7f160815eae0 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.745+0000 7f160dfe6640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f16080d1ec0 con 0x7f16080d2360 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160815e700 0x7f16081f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160815e700 0x7f16081f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35930/0 (socket says 172.21.15.12:35930) 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 -- 172.21.15.12:0/2609573531 learned_addr learned my addr 172.21.15.12:0/2609573531 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1607fff640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f160815eae0 0x7f16081f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f16077fe640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f16080d2360 0x7f1608161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 -- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f16080d2360 msgr2=0x7f1608161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f16080d2360 0x7f1608161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 -- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f160815eae0 msgr2=0x7f16081f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f160815eae0 0x7f16081f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 -- 172.21.15.12:0/2609573531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160815fd70 con 0x7f160815e700 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1607fff640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f160815eae0 0x7f16081f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f1606ffd640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160815e700 0x7f16081f7560 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f15fc02b7d0 tx=0x7f15fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f15effff640 1 -- 172.21.15.12:0/2609573531 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc041020 con 0x7f160815e700 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f15effff640 1 -- 172.21.15.12:0/2609573531 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15fc03c870 con 0x7f160815e700 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1608160050 con 0x7f160815e700 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f15effff640 1 -- 172.21.15.12:0/2609573531 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc037040 con 0x7f160815e700 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.746+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1608160610 con 0x7f160815e700 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.747+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16080689e0 con 0x7f160815e700 2024-04-15T15:23:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.748+0000 7f15effff640 1 -- 172.21.15.12:0/2609573531 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f15fc044020 con 0x7f160815e700 2024-04-15T15:23:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.748+0000 7f15effff640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f15f406c980 0x7f15f406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.748+0000 7f15effff640 1 -- 172.21.15.12:0/2609573531 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f15fc0ad850 con 0x7f160815e700 2024-04-15T15:23:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.748+0000 7f16077fe640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f15f406c980 0x7f15f406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.749+0000 7f16077fe640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f15f406c980 0x7f15f406ee40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f16080d4550 tx=0x7f15f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:21.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.753+0000 7f15effff640 1 -- 172.21.15.12:0/2609573531 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f15fc04b030 con 0x7f160815e700 2024-04-15T15:23:21.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.936+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f16080ba500 con 0x7f160815e700 2024-04-15T15:23:21.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.936+0000 7f15effff640 1 -- 172.21.15.12:0/2609573531 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f15fc07ba10 con 0x7f160815e700 2024-04-15T15:23:21.939 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:23:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f15f406c980 msgr2=0x7f15f406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f15f406c980 0x7f15f406ee40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f16080d4550 tx=0x7f15f8009040 comp rx=0 tx=0).stop 2024-04-15T15:23:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160815e700 msgr2=0x7f16081f7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160815e700 0x7f16081f7560 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f15fc02b7d0 tx=0x7f15fc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 shutdown_connections 2024-04-15T15:23:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f15f406c980 0x7f15f406ee40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f160815eae0 0x7f16081f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f160815e700 0x7f16081f7560 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 --2- 172.21.15.12:0/2609573531 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f16080d2360 0x7f1608161a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:21.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.938+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 >> 172.21.15.12:0/2609573531 conn(0x7f160808bf60 msgr2=0x7f16080d1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:21.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.939+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 shutdown_connections 2024-04-15T15:23:21.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:21.939+0000 7f160dfe6640 1 -- 172.21.15.12:0/2609573531 wait complete. 2024-04-15T15:23:21.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:23:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.055+0000 7fdf83f0d640 1 Processor -- start 2024-04-15T15:23:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.055+0000 7fdf83f0d640 1 -- start start 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.056+0000 7fdf83f0d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c15e990 0x7fdf7c06e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.056+0000 7fdf83f0d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7c06e9d0 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.056+0000 7fdf83f0d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7c06d3f0 con 0x7fdf7c09a9d0 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.056+0000 7fdf83f0d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7c15ed70 con 0x7fdf7c15e990 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.056+0000 7fdf814a8640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdf7c0b4ab0 0x7fdf7c09b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47244/0 (socket says 172.21.15.12:47244) 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.056+0000 7fdf814a8640 1 -- 172.21.15.12:0/1818946699 learned_addr learned my addr 172.21.15.12:0/1818946699 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.056+0000 7fdf81ca9640 1 --2- 172.21.15.12:0/1818946699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c15e990 0x7fdf7c06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.056+0000 7fdf81ca9640 1 -- 172.21.15.12:0/1818946699 >> v1:172.21.15.55:6789/0 conn(0x7fdf7c09a9d0 legacy=0x7fdf7c0cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.057+0000 7fdf81ca9640 1 -- 172.21.15.12:0/1818946699 >> v1:172.21.15.12:6789/0 conn(0x7fdf7c0b4ab0 legacy=0x7fdf7c09b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.057+0000 7fdf81ca9640 1 -- 172.21.15.12:0/1818946699 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c06d6d0 con 0x7fdf7c15e990 2024-04-15T15:23:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.057+0000 7fdf81ca9640 1 --2- 172.21.15.12:0/1818946699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c15e990 0x7fdf7c06e490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf740097d0 tx=0x7fdf7402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4b5c2db7f4410a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf80ca7640 1 -- 172.21.15.12:0/1818946699 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74035020 con 0x7fdf7c15e990 2024-04-15T15:23:22.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf80ca7640 1 -- 172.21.15.12:0/1818946699 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf74031890 con 0x7fdf7c15e990 2024-04-15T15:23:22.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf80ca7640 1 -- 172.21.15.12:0/1818946699 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74031c60 con 0x7fdf7c15e990 2024-04-15T15:23:22.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf83f0d640 1 -- 172.21.15.12:0/1818946699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c15e990 msgr2=0x7fdf7c06e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf83f0d640 1 --2- 172.21.15.12:0/1818946699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c15e990 0x7fdf7c06e490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf740097d0 tx=0x7fdf7402b930 comp rx=0 tx=0).stop 2024-04-15T15:23:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf83f0d640 1 -- 172.21.15.12:0/1818946699 shutdown_connections 2024-04-15T15:23:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf83f0d640 1 --2- 172.21.15.12:0/1818946699 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c15e990 0x7fdf7c06e490 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf83f0d640 1 -- 172.21.15.12:0/1818946699 >> 172.21.15.12:0/1818946699 conn(0x7fdf7c0b3eb0 msgr2=0x7fdf7c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf83f0d640 1 -- 172.21.15.12:0/1818946699 shutdown_connections 2024-04-15T15:23:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.065+0000 7fdf83f0d640 1 -- 172.21.15.12:0/1818946699 wait complete. 2024-04-15T15:23:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.066+0000 7fdf83f0d640 1 Processor -- start 2024-04-15T15:23:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.066+0000 7fdf83f0d640 1 -- start start 2024-04-15T15:23:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf83f0d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c09a9d0 0x7fdf7c14a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf83f0d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c0b4ab0 0x7fdf7c14bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf83f0d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c15e990 0x7fdf7c1fc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf83f0d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c06dc90 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf83f0d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c06e000 con 0x7fdf7c15e990 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf83f0d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7c0cd120 con 0x7fdf7c09a9d0 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf814a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c0b4ab0 0x7fdf7c14bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf814a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c0b4ab0 0x7fdf7c14bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35936/0 (socket says 172.21.15.12:35936) 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf814a8640 1 -- 172.21.15.12:0/519322924 learned_addr learned my addr 172.21.15.12:0/519322924 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf81ca9640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c09a9d0 0x7fdf7c14a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf824aa640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c15e990 0x7fdf7c1fc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf814a8640 1 -- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c09a9d0 msgr2=0x7fdf7c14a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf814a8640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c09a9d0 0x7fdf7c14a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf814a8640 1 -- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c15e990 msgr2=0x7fdf7c1fc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf814a8640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c15e990 0x7fdf7c1fc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.067+0000 7fdf814a8640 1 -- 172.21.15.12:0/519322924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c14acd0 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.068+0000 7fdf814a8640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c0b4ab0 0x7fdf7c14bd70 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000c4a0 tx=0x7fdf7000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.068+0000 7fdf7a7fc640 1 -- 172.21.15.12:0/519322924 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70017070 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.068+0000 7fdf7a7fc640 1 -- 172.21.15.12:0/519322924 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf70005ce0 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.068+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c14afb0 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.068+0000 7fdf7a7fc640 1 -- 172.21.15.12:0/519322924 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70007270 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.068+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c14b570 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.069+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf7c073f70 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.069+0000 7fdf7a7fc640 1 -- 172.21.15.12:0/519322924 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fdf7001a070 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.070+0000 7fdf7a7fc640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf6806c980 0x7fdf6806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.070+0000 7fdf7a7fc640 1 -- 172.21.15.12:0/519322924 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fdf7008af40 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.070+0000 7fdf81ca9640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf6806c980 0x7fdf6806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.070+0000 7fdf81ca9640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf6806c980 0x7fdf6806ee40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdf74036000 tx=0x7fdf7403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.074+0000 7fdf7a7fc640 1 -- 172.21.15.12:0/519322924 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdf7c073f70 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.270+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdf7c0cd500 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.271+0000 7fdf7a7fc640 1 -- 172.21.15.12:0/519322924 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7fdf7c0cd500 con 0x7fdf7c0b4ab0 2024-04-15T15:23:22.273 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.55:6837/3388147599 2024-04-16T15:23:15.001839+0000 2024-04-15T15:23:22.273 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.55:6835/3388147599 2024-04-16T15:23:15.001839+0000 2024-04-15T15:23:22.274 INFO:teuthology.orchestra.run.smithi012.stderr:listed 2 entries 2024-04-15T15:23:22.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf6806c980 msgr2=0x7fdf6806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf6806c980 0x7fdf6806ee40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdf74036000 tx=0x7fdf7403c040 comp rx=0 tx=0).stop 2024-04-15T15:23:22.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c0b4ab0 msgr2=0x7fdf7c14bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c0b4ab0 0x7fdf7c14bd70 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000c4a0 tx=0x7fdf7000c960 comp rx=0 tx=0).stop 2024-04-15T15:23:22.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 shutdown_connections 2024-04-15T15:23:22.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf6806c980 0x7fdf6806ee40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c15e990 0x7fdf7c1fc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf7c0b4ab0 0x7fdf7c14bd70 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 --2- 172.21.15.12:0/519322924 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf7c09a9d0 0x7fdf7c14a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.272+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 >> 172.21.15.12:0/519322924 conn(0x7fdf7c0b3eb0 msgr2=0x7fdf7c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:22.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.273+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 shutdown_connections 2024-04-15T15:23:22.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.273+0000 7fdf83f0d640 1 -- 172.21.15.12:0/519322924 wait complete. 2024-04-15T15:23:22.285 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-04-15T15:23:22.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:22.286 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:23:22.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[55627]: fuse finished with error 0 and tester_r 0 2024-04-15T15:23:22.314 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:23:22.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:22.315 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:23:22.482 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:23:22.482 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:23:22.482 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:22.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:23:22.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.625+0000 7f0a307a8640 1 Processor -- start 2024-04-15T15:23:22.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.625+0000 7f0a307a8640 1 -- start start 2024-04-15T15:23:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a307a8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a280b3160 0x7f0a2815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a307a8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0a280ae280 0x7f0a280ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a307a8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a2815cb50 con 0x7f0a280d0690 2024-04-15T15:23:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a307a8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0a2815bfd0 con 0x7f0a280b3160 2024-04-15T15:23:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a307a8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0a2815c2b0 con 0x7f0a280ae280 2024-04-15T15:23:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a2dd43640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a280b3160 0x7f0a2815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a2e544640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0a280d0690 0x7f0a280d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47012/0 (socket says 172.21.15.12:47012) 2024-04-15T15:23:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a2dd43640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a280b3160 0x7f0a2815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45920/0 (socket says 172.21.15.12:45920) 2024-04-15T15:23:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a2e544640 1 -- 172.21.15.12:0/641078303 learned_addr learned my addr 172.21.15.12:0/641078303 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a2ed45640 1 --2- 172.21.15.12:0/641078303 >> v2:172.21.15.55:3300/0 conn(0x7f0a280ae280 0x7f0a280ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a2dd43640 1 -- 172.21.15.12:0/641078303 >> v2:172.21.15.55:3300/0 conn(0x7f0a280ae280 msgr2=0x7f0a280ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a2dd43640 1 --2- 172.21.15.12:0/641078303 >> v2:172.21.15.55:3300/0 conn(0x7f0a280ae280 0x7f0a280ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.627+0000 7f0a2dd43640 1 -- 172.21.15.12:0/641078303 >> v1:172.21.15.12:6789/0 conn(0x7f0a280d0690 legacy=0x7f0a280d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.627+0000 7f0a2dd43640 1 -- 172.21.15.12:0/641078303 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2815c590 con 0x7f0a280b3160 2024-04-15T15:23:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.626+0000 7f0a2d542640 1 -- 172.21.15.12:0/641078303 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2434053309 0 0) 0x7f0a2815cb50 con 0x7f0a280d0690 2024-04-15T15:23:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.627+0000 7f0a2ed45640 1 --2- 172.21.15.12:0/641078303 >> v2:172.21.15.55:3300/0 conn(0x7f0a280ae280 0x7f0a280ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.627+0000 7f0a2dd43640 1 --2- 172.21.15.12:0/641078303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280b3160 0x7f0a2815b690 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0a280b1df0 tx=0x7f0a1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731723bb9bfb04ff server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.627+0000 7f0a2d542640 1 -- 172.21.15.12:0/641078303 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c035020 con 0x7f0a280b3160 2024-04-15T15:23:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.627+0000 7f0a2d542640 1 -- 172.21.15.12:0/641078303 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a1c031830 con 0x7f0a280b3160 2024-04-15T15:23:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.627+0000 7f0a2d542640 1 -- 172.21.15.12:0/641078303 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c031c00 con 0x7f0a280b3160 2024-04-15T15:23:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.628+0000 7f0a307a8640 1 -- 172.21.15.12:0/641078303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280b3160 msgr2=0x7f0a2815b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.628+0000 7f0a307a8640 1 --2- 172.21.15.12:0/641078303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280b3160 0x7f0a2815b690 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0a280b1df0 tx=0x7f0a1c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.628+0000 7f0a307a8640 1 -- 172.21.15.12:0/641078303 shutdown_connections 2024-04-15T15:23:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.628+0000 7f0a307a8640 1 --2- 172.21.15.12:0/641078303 >> v2:172.21.15.55:3300/0 conn(0x7f0a280ae280 0x7f0a280ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.628+0000 7f0a307a8640 1 --2- 172.21.15.12:0/641078303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280b3160 0x7f0a2815b690 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.628+0000 7f0a307a8640 1 -- 172.21.15.12:0/641078303 >> 172.21.15.12:0/641078303 conn(0x7f0a2806be00 msgr2=0x7f0a2806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:22.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.628+0000 7f0a307a8640 1 -- 172.21.15.12:0/641078303 shutdown_connections 2024-04-15T15:23:22.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.628+0000 7f0a307a8640 1 -- 172.21.15.12:0/641078303 wait complete. 2024-04-15T15:23:22.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.629+0000 7f0a307a8640 1 Processor -- start 2024-04-15T15:23:22.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.629+0000 7f0a307a8640 1 -- start start 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a307a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280ae280 0x7f0a2806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a307a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a280b3160 0x7f0a28145880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a307a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a280d0690 0x7f0a281506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a307a8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2815bbd0 con 0x7f0a280ae280 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a307a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0a280b4fb0 con 0x7f0a280b3160 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a307a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0a2815bd50 con 0x7f0a280d0690 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2e544640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280ae280 0x7f0a2806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2e544640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280ae280 0x7f0a2806ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45930/0 (socket says 172.21.15.12:45930) 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2e544640 1 -- 172.21.15.12:0/2195586880 learned_addr learned my addr 172.21.15.12:0/2195586880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:22.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2dd43640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a280b3160 0x7f0a28145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2ed45640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a280d0690 0x7f0a281506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2e544640 1 -- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a280d0690 msgr2=0x7f0a281506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2e544640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a280d0690 0x7f0a281506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2e544640 1 -- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a280b3160 msgr2=0x7f0a28145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2e544640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a280b3160 0x7f0a28145880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.630+0000 7f0a2e544640 1 -- 172.21.15.12:0/2195586880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a280cf780 con 0x7f0a280ae280 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.631+0000 7f0a2e544640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280ae280 0x7f0a2806ea80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000c430 tx=0x7f0a2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.631+0000 7f0a26ffd640 1 -- 172.21.15.12:0/2195586880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200074e0 con 0x7f0a280ae280 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.631+0000 7f0a26ffd640 1 -- 172.21.15.12:0/2195586880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a200047e0 con 0x7f0a280ae280 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.631+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2814fbe0 con 0x7f0a280ae280 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.631+0000 7f0a26ffd640 1 -- 172.21.15.12:0/2195586880 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20004bd0 con 0x7f0a280ae280 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.631+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a281501a0 con 0x7f0a280ae280 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.632+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a280689e0 con 0x7f0a280ae280 2024-04-15T15:23:22.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.633+0000 7f0a26ffd640 1 -- 172.21.15.12:0/2195586880 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0a20011070 con 0x7f0a280ae280 2024-04-15T15:23:22.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.633+0000 7f0a26ffd640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a1406c9b0 0x7f0a1406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.633+0000 7f0a26ffd640 1 -- 172.21.15.12:0/2195586880 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0a2008b9a0 con 0x7f0a280ae280 2024-04-15T15:23:22.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.633+0000 7f0a2dd43640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a1406c9b0 0x7f0a1406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.634+0000 7f0a2dd43640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a1406c9b0 0x7f0a1406ee70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c036000 tx=0x7f0a1c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.638+0000 7f0a26ffd640 1 -- 172.21.15.12:0/2195586880 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a2000e280 con 0x7f0a280ae280 2024-04-15T15:23:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.824+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a280739e0 con 0x7f0a280ae280 2024-04-15T15:23:22.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.825+0000 7f0a26ffd640 1 -- 172.21.15.12:0/2195586880 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17084 (secure 0 0 0) 0x7f0a20059be0 con 0x7f0a280ae280 2024-04-15T15:23:22.828 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:22.828 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":42,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:23:15.001862+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6837/3388147599":"2024-04-16T15:23:15.001839+0000","172.21.15.55:6835/3388147599":"2024-04-16T15:23:15.001839+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:23:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a1406c9b0 msgr2=0x7f0a1406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a1406c9b0 0x7f0a1406ee70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c036000 tx=0x7f0a1c008030 comp rx=0 tx=0).stop 2024-04-15T15:23:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280ae280 msgr2=0x7f0a2806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280ae280 0x7f0a2806ea80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000c430 tx=0x7f0a2000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 shutdown_connections 2024-04-15T15:23:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0a280d0690 0x7f0a281506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a1406c9b0 0x7f0a1406ee70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a280b3160 0x7f0a28145880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 --2- 172.21.15.12:0/2195586880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280ae280 0x7f0a2806ea80 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.827+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 >> 172.21.15.12:0/2195586880 conn(0x7f0a2806be00 msgr2=0x7f0a280b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.828+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 shutdown_connections 2024-04-15T15:23:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.828+0000 7f0a307a8640 1 -- 172.21.15.12:0/2195586880 wait complete. 2024-04-15T15:23:22.841 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:23:22.841 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:23:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.946+0000 7f1fcef96640 1 Processor -- start 2024-04-15T15:23:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.946+0000 7f1fcef96640 1 -- start start 2024-04-15T15:23:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fcef96640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1fc80b4e90 0x7f1fc8169880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fcef96640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc80b6960 con 0x7f1fc815ea40 2024-04-15T15:23:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fcef96640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc80cff70 con 0x7f1fc80d3480 2024-04-15T15:23:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fcef96640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1fc80d0250 con 0x7f1fc80b4e90 2024-04-15T15:23:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fccd32640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1fc815ea40 0x7f1fc80b5b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47020/0 (socket says 172.21.15.12:47020) 2024-04-15T15:23:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fccd32640 1 -- 172.21.15.12:0/3171897985 learned_addr learned my addr 172.21.15.12:0/3171897985 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fcd533640 1 --2- 172.21.15.12:0/3171897985 >> v2:172.21.15.55:3300/0 conn(0x7f1fc80b4e90 0x7f1fc8169880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696306335 0 0) 0x7f1fc80b6960 con 0x7f1fc815ea40 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb0003520 con 0x7f1fc815ea40 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2920507452 0 0) 0x7f1fb0003520 con 0x7f1fc815ea40 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 >> v2:172.21.15.55:3300/0 conn(0x7f1fc80b4e90 msgr2=0x7f1fc8169880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fbf7fe640 1 --2- 172.21.15.12:0/3171897985 >> v2:172.21.15.55:3300/0 conn(0x7f1fc80b4e90 0x7f1fc8169880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 >> v1:172.21.15.55:6789/0 conn(0x7f1fc80d3480 legacy=0x7f1fc80b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.947+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc80d0530 con 0x7f1fc815ea40 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f1fc40019f0 con 0x7f1fc815ea40 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 >> v1:172.21.15.12:6789/0 conn(0x7f1fc815ea40 legacy=0x7f1fc80b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbf7fe640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fb00039b0 0x7f1fb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbf7fe640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0004800 0x7f1fb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbf7fe640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb00051c0 0x7f1fb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc80d0250 con 0x7f1fb0004800 2024-04-15T15:23:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb000c070 con 0x7f1fb00051c0 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1fb000c270 con 0x7f1fb00039b0 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbffff640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0004800 0x7f1fb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fcd533640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb00051c0 0x7f1fb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fccd32640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fb00039b0 0x7f1fb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbffff640 1 -- 172.21.15.12:0/3171897985 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fb00039b0 msgr2=0x7f1fb0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbffff640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fb00039b0 0x7f1fb0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbffff640 1 -- 172.21.15.12:0/3171897985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb00051c0 msgr2=0x7f1fb000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbffff640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb00051c0 0x7f1fb000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fcd533640 1 -- 172.21.15.12:0/3171897985 reap_dead start 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.948+0000 7f1fbffff640 1 -- 172.21.15.12:0/3171897985 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb0003520 con 0x7f1fb0004800 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fbffff640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0004800 0x7f1fb0004c80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8001750 tx=0x7f1fb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba9f90519934e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8001370 con 0x7f1fb0004800 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fb80052d0 con 0x7f1fb0004800 2024-04-15T15:23:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fbf7fe640 1 -- 172.21.15.12:0/3171897985 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8003830 con 0x7f1fb0004800 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fcef96640 1 -- 172.21.15.12:0/3171897985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0004800 msgr2=0x7f1fb0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fcef96640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0004800 0x7f1fb0004c80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8001750 tx=0x7f1fb0005830 comp rx=0 tx=0).stop 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fcef96640 1 -- 172.21.15.12:0/3171897985 shutdown_connections 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fcef96640 1 --2- 172.21.15.12:0/3171897985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fb0004800 0x7f1fb0004c80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.949+0000 7f1fcef96640 1 -- 172.21.15.12:0/3171897985 >> 172.21.15.12:0/3171897985 conn(0x7f1fc808bf60 msgr2=0x7f1fc80d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.950+0000 7f1fcef96640 1 -- 172.21.15.12:0/3171897985 shutdown_connections 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.950+0000 7f1fcef96640 1 -- 172.21.15.12:0/3171897985 wait complete. 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.951+0000 7f1fcef96640 1 Processor -- start 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.951+0000 7f1fcef96640 1 -- start start 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.951+0000 7f1fcef96640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb0004800 0x7f1fc8153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcef96640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fc8153ad0 0x7f1fc80b9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcef96640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc81e86a0 0x7f1fc81e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcef96640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc80b38c0 con 0x7f1fc81e86a0 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcef96640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc80d0af0 con 0x7f1fb0004800 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcef96640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc80b3a40 con 0x7f1fc8153ad0 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc81e86a0 0x7f1fc81e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc81e86a0 0x7f1fc81e8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45938/0 (socket says 172.21.15.12:45938) 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 -- 172.21.15.12:0/1891630504 learned_addr learned my addr 172.21.15.12:0/1891630504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fccd32640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb0004800 0x7f1fc8153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fbffff640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fc8153ad0 0x7f1fc80b9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 -- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fc8153ad0 msgr2=0x7f1fc80b9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fc8153ad0 0x7f1fc80b9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 -- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb0004800 msgr2=0x7f1fc8153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb0004800 0x7f1fc8153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 -- 172.21.15.12:0/1891630504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc80cb640 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcd533640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc81e86a0 0x7f1fc81e8b20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80b5320 tx=0x7f1fc0011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fbd7fa640 1 -- 172.21.15.12:0/1891630504 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0004030 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fbd7fa640 1 -- 172.21.15.12:0/1891630504 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc001b070 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc80b7760 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fbd7fa640 1 -- 172.21.15.12:0/1891630504 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0004430 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.952+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc80b7d20 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.953+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc8153310 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.954+0000 7f1fbd7fa640 1 -- 172.21.15.12:0/1891630504 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f1fc0012040 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.954+0000 7f1fbd7fa640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1fb406c980 0x7f1fb406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.954+0000 7f1fbd7fa640 1 -- 172.21.15.12:0/1891630504 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f1fc008bbe0 con 0x7f1fc81e86a0 2024-04-15T15:23:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.955+0000 7f1fccd32640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1fb406c980 0x7f1fb406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.955+0000 7f1fccd32640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1fb406c980 0x7f1fb406ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4001560 tx=0x7f1fc4006040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:22.959+0000 7f1fbd7fa640 1 -- 172.21.15.12:0/1891630504 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1fc8153310 con 0x7f1fc81e86a0 2024-04-15T15:23:23.081 INFO:tasks.ceph.mds.b.smithi055.stderr:2024-04-15T15:23:23.080+0000 7f1eaa5e8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:23:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.143+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1fc80689e0 con 0x7f1fc81e86a0 2024-04-15T15:23:23.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.143+0000 7f1fbd7fa640 1 -- 172.21.15.12:0/1891630504 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f1fc0059e20 con 0x7f1fc81e86a0 2024-04-15T15:23:23.146 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1fb406c980 msgr2=0x7f1fb406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1fb406c980 0x7f1fb406ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4001560 tx=0x7f1fc4006040 comp rx=0 tx=0).stop 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc81e86a0 msgr2=0x7f1fc81e8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc81e86a0 0x7f1fc81e8b20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80b5320 tx=0x7f1fc0011cf0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 shutdown_connections 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fc81e86a0 0x7f1fc81e8b20 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1fc8153ad0 0x7f1fc80b9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f1fb406c980 0x7f1fb406ee40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 --2- 172.21.15.12:0/1891630504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fb0004800 0x7f1fc8153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 >> 172.21.15.12:0/1891630504 conn(0x7f1fc808bf60 msgr2=0x7f1fc806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 shutdown_connections 2024-04-15T15:23:23.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.145+0000 7f1fcef96640 1 -- 172.21.15.12:0/1891630504 wait complete. 2024-04-15T15:23:23.159 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:23:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.264+0000 7fe9c4de9640 1 Processor -- start 2024-04-15T15:23:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.264+0000 7fe9c4de9640 1 -- start start 2024-04-15T15:23:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9c4de9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9c00d0690 0x7fe9c00d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9c4de9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe9c00ae280 0x7fe9c00ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9c4de9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9c00d0fb0 con 0x7fe9c00b3160 2024-04-15T15:23:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9c4de9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe9c015bda0 con 0x7fe9c00d0690 2024-04-15T15:23:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9c4de9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe9c015cb50 con 0x7fe9c00ae280 2024-04-15T15:23:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9be59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9c00d0690 0x7fe9c00d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9bdd9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe9c00b3160 0x7fe9c015b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47024/0 (socket says 172.21.15.12:47024) 2024-04-15T15:23:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9be59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9c00d0690 0x7fe9c00d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45948/0 (socket says 172.21.15.12:45948) 2024-04-15T15:23:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9bdd9b640 1 -- 172.21.15.12:0/4201761871 learned_addr learned my addr 172.21.15.12:0/4201761871 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9bed9d640 1 --2- 172.21.15.12:0/4201761871 >> v2:172.21.15.55:3300/0 conn(0x7fe9c00ae280 0x7fe9c00ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9be59c640 1 -- 172.21.15.12:0/4201761871 >> v2:172.21.15.55:3300/0 conn(0x7fe9c00ae280 msgr2=0x7fe9c00ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9be59c640 1 --2- 172.21.15.12:0/4201761871 >> v2:172.21.15.55:3300/0 conn(0x7fe9c00ae280 0x7fe9c00ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9be59c640 1 -- 172.21.15.12:0/4201761871 >> v1:172.21.15.12:6789/0 conn(0x7fe9c00b3160 legacy=0x7fe9c015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.265+0000 7fe9be59c640 1 -- 172.21.15.12:0/4201761871 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c015bfd0 con 0x7fe9c00d0690 2024-04-15T15:23:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.266+0000 7fe9bed9d640 1 --2- 172.21.15.12:0/4201761871 >> v2:172.21.15.55:3300/0 conn(0x7fe9c00ae280 0x7fe9c00ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.266+0000 7fe9be59c640 1 --2- 172.21.15.12:0/4201761871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c00d0a70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe9c006ffe0 tx=0x7fe9b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda6cc2c6cdd82 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.266+0000 7fe9bd59a640 1 -- 172.21.15.12:0/4201761871 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8035020 con 0x7fe9c00d0690 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.266+0000 7fe9bd59a640 1 -- 172.21.15.12:0/4201761871 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9b8031830 con 0x7fe9c00d0690 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.266+0000 7fe9bd59a640 1 -- 172.21.15.12:0/4201761871 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8031c00 con 0x7fe9c00d0690 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.267+0000 7fe9c4de9640 1 -- 172.21.15.12:0/4201761871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 msgr2=0x7fe9c00d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.267+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/4201761871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c00d0a70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe9c006ffe0 tx=0x7fe9b802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.267+0000 7fe9c4de9640 1 -- 172.21.15.12:0/4201761871 shutdown_connections 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.267+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/4201761871 >> v2:172.21.15.55:3300/0 conn(0x7fe9c00ae280 0x7fe9c00ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.267+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/4201761871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c00d0a70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.267+0000 7fe9c4de9640 1 -- 172.21.15.12:0/4201761871 >> 172.21.15.12:0/4201761871 conn(0x7fe9c006be00 msgr2=0x7fe9c006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.267+0000 7fe9c4de9640 1 -- 172.21.15.12:0/4201761871 shutdown_connections 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.267+0000 7fe9c4de9640 1 -- 172.21.15.12:0/4201761871 wait complete. 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.268+0000 7fe9c4de9640 1 Processor -- start 2024-04-15T15:23:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.268+0000 7fe9c4de9640 1 -- start start 2024-04-15T15:23:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9c4de9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9c00ae280 0x7fe9c00cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9c4de9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9c00b3160 0x7fe9c01e6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9c4de9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c0154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9c4de9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c015c870 con 0x7fe9c00d0690 2024-04-15T15:23:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9c4de9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c015c590 con 0x7fe9c00ae280 2024-04-15T15:23:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9c4de9640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c00b4fb0 con 0x7fe9c00b3160 2024-04-15T15:23:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9bed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c0154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9bed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c0154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45960/0 (socket says 172.21.15.12:45960) 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9bed9d640 1 -- 172.21.15.12:0/74138412 learned_addr learned my addr 172.21.15.12:0/74138412 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9be59c640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9c00ae280 0x7fe9c00cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9bdd9b640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9c00b3160 0x7fe9c01e6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.269+0000 7fe9bed9d640 1 -- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9c00b3160 msgr2=0x7fe9c01e6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9bed9d640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9c00b3160 0x7fe9c01e6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9bed9d640 1 -- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9c00ae280 msgr2=0x7fe9c00cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9bed9d640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9c00ae280 0x7fe9c00cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9bed9d640 1 -- 172.21.15.12:0/74138412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c00cd830 con 0x7fe9c00d0690 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9be59c640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9c00ae280 0x7fe9c00cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9bed9d640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c0154a70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0007df0 tx=0x7fe9b000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9b6ffd640 1 -- 172.21.15.12:0/74138412 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b001a070 con 0x7fe9c00d0690 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9b6ffd640 1 -- 172.21.15.12:0/74138412 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9b0003990 con 0x7fe9c00d0690 2024-04-15T15:23:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c00cdb10 con 0x7fe9c00d0690 2024-04-15T15:23:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9b6ffd640 1 -- 172.21.15.12:0/74138412 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0003d80 con 0x7fe9c00d0690 2024-04-15T15:23:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.270+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c00ce0d0 con 0x7fe9c00d0690 2024-04-15T15:23:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.271+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c00690f0 con 0x7fe9c00d0690 2024-04-15T15:23:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.271+0000 7fe9b6ffd640 1 -- 172.21.15.12:0/74138412 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe9b001d070 con 0x7fe9c00d0690 2024-04-15T15:23:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.272+0000 7fe9b6ffd640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe9a006c980 0x7fe9a006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.272+0000 7fe9b6ffd640 1 -- 172.21.15.12:0/74138412 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fe9b008c0c0 con 0x7fe9c00d0690 2024-04-15T15:23:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.272+0000 7fe9be59c640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe9a006c980 0x7fe9a006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.273+0000 7fe9be59c640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe9a006c980 0x7fe9a006ee40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8036000 tx=0x7fe9b8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.276+0000 7fe9b6ffd640 1 -- 172.21.15.12:0/74138412 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9c00690f0 con 0x7fe9c00d0690 2024-04-15T15:23:23.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.460+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe9c00689e0 con 0x7fe9c00d0690 2024-04-15T15:23:23.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.460+0000 7fe9b6ffd640 1 -- 172.21.15.12:0/74138412 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7fe9b0010070 con 0x7fe9c00d0690 2024-04-15T15:23:23.462 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:23:23.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe9a006c980 msgr2=0x7fe9a006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe9a006c980 0x7fe9a006ee40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8036000 tx=0x7fe9b8008030 comp rx=0 tx=0).stop 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 msgr2=0x7fe9c0154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c0154a70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0007df0 tx=0x7fe9b000d4a0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 shutdown_connections 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9c00d0690 0x7fe9c0154a70 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe9c00b3160 0x7fe9c01e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe9a006c980 0x7fe9a006ee40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 --2- 172.21.15.12:0/74138412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9c00ae280 0x7fe9c00cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 >> 172.21.15.12:0/74138412 conn(0x7fe9c006be00 msgr2=0x7fe9c00b3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 shutdown_connections 2024-04-15T15:23:23.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.462+0000 7fe9c4de9640 1 -- 172.21.15.12:0/74138412 wait complete. 2024-04-15T15:23:23.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:23:23.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.581+0000 7f608cd9d640 1 Processor -- start 2024-04-15T15:23:23.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.581+0000 7f608cd9d640 1 -- start start 2024-04-15T15:23:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f608cd9d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d0d40 0x7f60880cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f608cd9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60880d4e60 0x7f60880d1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f608cd9d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60880d1660 con 0x7f60880d2cc0 2024-04-15T15:23:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f608cd9d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f60880d1a00 con 0x7f60880d4e60 2024-04-15T15:23:23.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f608cd9d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f60880d1f90 con 0x7f60880d0d40 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f607effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60880d4e60 0x7f60880d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f607effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60880d4e60 0x7f60880d1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45968/0 (socket says 172.21.15.12:45968) 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f607effd640 1 -- 172.21.15.12:0/4268329466 learned_addr learned my addr 172.21.15.12:0/4268329466 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f60877fe640 1 --2- 172.21.15.12:0/4268329466 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d0d40 0x7f60880cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f607effd640 1 -- 172.21.15.12:0/4268329466 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d0d40 msgr2=0x7f60880cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f607effd640 1 --2- 172.21.15.12:0/4268329466 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d0d40 0x7f60880cfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f607effd640 1 -- 172.21.15.12:0/4268329466 >> v1:172.21.15.55:6789/0 conn(0x7f60880d2cc0 legacy=0x7f60880b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.582+0000 7f607effd640 1 -- 172.21.15.12:0/4268329466 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60880d02e0 con 0x7f60880d4e60 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.583+0000 7f607effd640 1 --2- 172.21.15.12:0/4268329466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d4e60 0x7f60880d1120 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6070009850 tx=0x7f607002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b07642d18669b08 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.583+0000 7f6086ffd640 1 -- 172.21.15.12:0/4268329466 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070037070 con 0x7f60880d4e60 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.583+0000 7f6086ffd640 1 -- 172.21.15.12:0/4268329466 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6070004030 con 0x7f60880d4e60 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.583+0000 7f6086ffd640 1 -- 172.21.15.12:0/4268329466 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070004460 con 0x7f60880d4e60 2024-04-15T15:23:23.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.584+0000 7f608cd9d640 1 -- 172.21.15.12:0/4268329466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d4e60 msgr2=0x7f60880d1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.584+0000 7f608cd9d640 1 --2- 172.21.15.12:0/4268329466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d4e60 0x7f60880d1120 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6070009850 tx=0x7f607002b840 comp rx=0 tx=0).stop 2024-04-15T15:23:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.584+0000 7f608cd9d640 1 -- 172.21.15.12:0/4268329466 shutdown_connections 2024-04-15T15:23:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.584+0000 7f608cd9d640 1 --2- 172.21.15.12:0/4268329466 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d4e60 0x7f60880d1120 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.584+0000 7f608cd9d640 1 --2- 172.21.15.12:0/4268329466 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d0d40 0x7f60880cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.584+0000 7f608cd9d640 1 -- 172.21.15.12:0/4268329466 >> 172.21.15.12:0/4268329466 conn(0x7f608808bf60 msgr2=0x7f60880d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.584+0000 7f608cd9d640 1 -- 172.21.15.12:0/4268329466 shutdown_connections 2024-04-15T15:23:23.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.584+0000 7f608cd9d640 1 -- 172.21.15.12:0/4268329466 wait complete. 2024-04-15T15:23:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.585+0000 7f608cd9d640 1 Processor -- start 2024-04-15T15:23:23.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.585+0000 7f608cd9d640 1 -- start start 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f608cd9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60880d0d40 0x7f60880ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f608cd9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d2cc0 0x7f60881616d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f608cd9d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d4e60 0x7f60880c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f608cd9d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f60880d27a0 con 0x7f60880d2cc0 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f608cd9d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f60880d24c0 con 0x7f60880d0d40 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f608cd9d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f608815fcc0 con 0x7f60880d4e60 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f607effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d2cc0 0x7f60881616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f607effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d2cc0 0x7f60881616d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45972/0 (socket says 172.21.15.12:45972) 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f607effd640 1 -- 172.21.15.12:0/696257708 learned_addr learned my addr 172.21.15.12:0/696257708 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f6087fff640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d4e60 0x7f60880c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f60877fe640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60880d0d40 0x7f60880ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f607effd640 1 -- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d4e60 msgr2=0x7f60880c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f607effd640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d4e60 0x7f60880c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f607effd640 1 -- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60880d0d40 msgr2=0x7f60880ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f607effd640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60880d0d40 0x7f60880ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.586+0000 7f607effd640 1 -- 172.21.15.12:0/696257708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60880b71f0 con 0x7f60880d2cc0 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.587+0000 7f607effd640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d2cc0 0x7f60881616d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6070035040 tx=0x7f60700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.587+0000 7f6084ff9640 1 -- 172.21.15.12:0/696257708 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070042020 con 0x7f60880d2cc0 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.587+0000 7f6084ff9640 1 -- 172.21.15.12:0/696257708 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6070038040 con 0x7f60880d2cc0 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.587+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60880b74d0 con 0x7f60880d2cc0 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.587+0000 7f6084ff9640 1 -- 172.21.15.12:0/696257708 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070037050 con 0x7f60880d2cc0 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.587+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60880b7a90 con 0x7f60880d2cc0 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.588+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f608804fd50 con 0x7f60880d2cc0 2024-04-15T15:23:23.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.588+0000 7f6084ff9640 1 -- 172.21.15.12:0/696257708 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6070003cc0 con 0x7f60880d2cc0 2024-04-15T15:23:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.589+0000 7f6084ff9640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f606806c9b0 0x7f606806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.589+0000 7f6084ff9640 1 -- 172.21.15.12:0/696257708 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f607002f030 con 0x7f60880d2cc0 2024-04-15T15:23:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.589+0000 7f60877fe640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f606806c9b0 0x7f606806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.590+0000 7f60877fe640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f606806c9b0 0x7f606806ee70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6078009aa0 tx=0x7f6078009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.593+0000 7f6084ff9640 1 -- 172.21.15.12:0/696257708 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f607007ad80 con 0x7f60880d2cc0 2024-04-15T15:23:23.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.780+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f60880d1b90 con 0x7f60880d2cc0 2024-04-15T15:23:23.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.780+0000 7f6084ff9640 1 -- 172.21.15.12:0/696257708 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f607007f080 con 0x7f60880d2cc0 2024-04-15T15:23:23.783 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:23:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f606806c9b0 msgr2=0x7f606806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f606806c9b0 0x7f606806ee70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6078009aa0 tx=0x7f6078009190 comp rx=0 tx=0).stop 2024-04-15T15:23:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d2cc0 msgr2=0x7f60881616d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d2cc0 0x7f60881616d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6070035040 tx=0x7f60700046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 shutdown_connections 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f60880d4e60 0x7f60880c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880d2cc0 0x7f60881616d0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f606806c9b0 0x7f606806ee70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 --2- 172.21.15.12:0/696257708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60880d0d40 0x7f60880ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 >> 172.21.15.12:0/696257708 conn(0x7f608808bf60 msgr2=0x7f608806d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 shutdown_connections 2024-04-15T15:23:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.783+0000 7f608cd9d640 1 -- 172.21.15.12:0/696257708 wait complete. 2024-04-15T15:23:23.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:23:23.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.899+0000 7fe496fe9640 1 Processor -- start 2024-04-15T15:23:23.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.899+0000 7fe496fe9640 1 -- start start 2024-04-15T15:23:23.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.900+0000 7fe496fe9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe49015e790 0x7fe4900b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.900+0000 7fe496fe9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe49015eb70 con 0x7fe4900b7be0 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.900+0000 7fe496fe9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4900d3c80 con 0x7fe4900d1cf0 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.900+0000 7fe496fe9640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe4900d3790 con 0x7fe49015e790 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.900+0000 7fe48ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe4900b7be0 0x7fe4900d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47026/0 (socket says 172.21.15.12:47026) 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.900+0000 7fe48ffff640 1 -- 172.21.15.12:0/3821373285 learned_addr learned my addr 172.21.15.12:0/3821373285 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.900+0000 7fe494d85640 1 --2- 172.21.15.12:0/3821373285 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe49015e790 0x7fe4900b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.900+0000 7fe48f7fe640 1 -- 172.21.15.12:0/3821373285 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404246010 0 0) 0x7fe49015eb70 con 0x7fe4900b7be0 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.901+0000 7fe48f7fe640 1 -- 172.21.15.12:0/3821373285 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe478003520 con 0x7fe4900b7be0 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.901+0000 7fe48f7fe640 1 -- 172.21.15.12:0/3821373285 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609984306 0 0) 0x7fe4900d3c80 con 0x7fe4900d1cf0 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.901+0000 7fe48f7fe640 1 -- 172.21.15.12:0/3821373285 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe49015eb70 con 0x7fe4900d1cf0 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.901+0000 7fe48f7fe640 1 -- 172.21.15.12:0/3821373285 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 223974845 0 0) 0x7fe478003520 con 0x7fe4900b7be0 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.901+0000 7fe494d85640 1 -- 172.21.15.12:0/3821373285 >> v1:172.21.15.55:6789/0 conn(0x7fe4900d1cf0 legacy=0x7fe4900d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.901+0000 7fe494d85640 1 -- 172.21.15.12:0/3821373285 >> v1:172.21.15.12:6789/0 conn(0x7fe4900b7be0 legacy=0x7fe4900d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.901+0000 7fe494d85640 1 -- 172.21.15.12:0/3821373285 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4900d3f60 con 0x7fe49015e790 2024-04-15T15:23:23.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe494d85640 1 --2- 172.21.15.12:0/3821373285 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe49015e790 0x7fe4900b76a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe4800097d0 tx=0x7fe48002b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7fc14ae6d76b3fa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe48f7fe640 1 -- 172.21.15.12:0/3821373285 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480037070 con 0x7fe49015e790 2024-04-15T15:23:23.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe48f7fe640 1 -- 172.21.15.12:0/3821373285 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe480030510 con 0x7fe49015e790 2024-04-15T15:23:23.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe48f7fe640 1 -- 172.21.15.12:0/3821373285 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe484001690 con 0x7fe49015e790 2024-04-15T15:23:23.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe496fe9640 1 -- 172.21.15.12:0/3821373285 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe49015e790 msgr2=0x7fe4900b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe496fe9640 1 --2- 172.21.15.12:0/3821373285 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe49015e790 0x7fe4900b76a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe4800097d0 tx=0x7fe48002b6e0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe496fe9640 1 -- 172.21.15.12:0/3821373285 shutdown_connections 2024-04-15T15:23:23.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe496fe9640 1 --2- 172.21.15.12:0/3821373285 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe49015e790 0x7fe4900b76a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe496fe9640 1 -- 172.21.15.12:0/3821373285 >> 172.21.15.12:0/3821373285 conn(0x7fe490067b40 msgr2=0x7fe490067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:23.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe496fe9640 1 -- 172.21.15.12:0/3821373285 shutdown_connections 2024-04-15T15:23:23.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.902+0000 7fe496fe9640 1 -- 172.21.15.12:0/3821373285 wait complete. 2024-04-15T15:23:23.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.903+0000 7fe496fe9640 1 Processor -- start 2024-04-15T15:23:23.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.904+0000 7fe496fe9640 1 -- start start 2024-04-15T15:23:23.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.904+0000 7fe496fe9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4900b7be0 0x7fe4900c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.904+0000 7fe496fe9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4900d1cf0 0x7fe49014ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.904+0000 7fe496fe9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe49015e790 0x7fe4901609a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.904+0000 7fe496fe9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4900d5150 con 0x7fe49015e790 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.904+0000 7fe496fe9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe4900d45b0 con 0x7fe4900b7be0 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.904+0000 7fe496fe9640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe4900d52d0 con 0x7fe4900d1cf0 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.904+0000 7fe495586640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe49015e790 0x7fe4901609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe495586640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe49015e790 0x7fe4901609a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45982/0 (socket says 172.21.15.12:45982) 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe495586640 1 -- 172.21.15.12:0/467069643 learned_addr learned my addr 172.21.15.12:0/467069643 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe494d85640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4900b7be0 0x7fe4900c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe48ffff640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4900d1cf0 0x7fe49014ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe495586640 1 -- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4900d1cf0 msgr2=0x7fe49014ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe495586640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4900d1cf0 0x7fe49014ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe495586640 1 -- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4900b7be0 msgr2=0x7fe4900c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe495586640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4900b7be0 0x7fe4900c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe495586640 1 -- 172.21.15.12:0/467069643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49014cf50 con 0x7fe49015e790 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe48ffff640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4900d1cf0 0x7fe49014ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:23.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe494d85640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4900b7be0 0x7fe4900c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe495586640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe49015e790 0x7fe4901609a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c730 tx=0x7fe48800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe48d7fa640 1 -- 172.21.15.12:0/467069643 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488003b70 con 0x7fe49015e790 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe48d7fa640 1 -- 172.21.15.12:0/467069643 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe488003d10 con 0x7fe49015e790 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe49015fbe0 con 0x7fe49015e790 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe48d7fa640 1 -- 172.21.15.12:0/467069643 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488005310 con 0x7fe49015e790 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.905+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4901601a0 con 0x7fe49015e790 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.906+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe490073c40 con 0x7fe49015e790 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.907+0000 7fe48d7fa640 1 -- 172.21.15.12:0/467069643 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe4880054b0 con 0x7fe49015e790 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.907+0000 7fe48d7fa640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe47c06c9b0 0x7fe47c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.907+0000 7fe48d7fa640 1 -- 172.21.15.12:0/467069643 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fe48808bbf0 con 0x7fe49015e790 2024-04-15T15:23:23.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.907+0000 7fe494d85640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe47c06c9b0 0x7fe47c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:23.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.908+0000 7fe494d85640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe47c06c9b0 0x7fe47c06ee70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe480034000 tx=0x7fe480008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:23.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:23.912+0000 7fe48d7fa640 1 -- 172.21.15.12:0/467069643 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe490073c40 con 0x7fe49015e790 2024-04-15T15:23:24.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.110+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe4900689e0 con 0x7fe49015e790 2024-04-15T15:23:24.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.110+0000 7fe48d7fa640 1 -- 172.21.15.12:0/467069643 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7fe488059e40 con 0x7fe49015e790 2024-04-15T15:23:24.113 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.55:6837/3388147599 2024-04-16T15:23:15.001839+0000 2024-04-15T15:23:24.114 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.55:6835/3388147599 2024-04-16T15:23:15.001839+0000 2024-04-15T15:23:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:listed 2 entries 2024-04-15T15:23:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe47c06c9b0 msgr2=0x7fe47c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe47c06c9b0 0x7fe47c06ee70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe480034000 tx=0x7fe480008030 comp rx=0 tx=0).stop 2024-04-15T15:23:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe49015e790 msgr2=0x7fe4901609a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe49015e790 0x7fe4901609a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c730 tx=0x7fe48800cbf0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 shutdown_connections 2024-04-15T15:23:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe49015e790 0x7fe4901609a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe4900d1cf0 0x7fe49014ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe47c06c9b0 0x7fe47c06ee70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 --2- 172.21.15.12:0/467069643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe4900b7be0 0x7fe4900c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 >> 172.21.15.12:0/467069643 conn(0x7fe490067b40 msgr2=0x7fe4900b9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 shutdown_connections 2024-04-15T15:23:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.112+0000 7fe496fe9640 1 -- 172.21.15.12:0/467069643 wait complete. 2024-04-15T15:23:24.125 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi055.front.sepia.ceph.com... 2024-04-15T15:23:24.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:24.126 DEBUG:teuthology.orchestra.run.smithi055:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:23:24.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi055.stderr:ceph-fuse[46132]: fuse finished with error 0 and tester_r 0 2024-04-15T15:23:24.160 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:23:24.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:24.160 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:23:24.333 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:23:24.333 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:23:24.333 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:24.351 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:23:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.494+0000 7f86681a8640 1 Processor -- start 2024-04-15T15:23:24.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.494+0000 7f86681a8640 1 -- start start 2024-04-15T15:23:24.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f86681a8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f866015e790 0x7f86600d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f86681a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 0x7f86600d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f86681a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86600d2480 con 0x7f86600cfd20 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f86681a8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f86600d2db0 con 0x7f866015e790 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f86681a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f86600d2760 con 0x7f86600d1610 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f8665f44640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f866015e790 0x7f86600d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f8665f44640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f866015e790 0x7f86600d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45996/0 (socket says 172.21.15.12:45996) 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f8665f44640 1 -- 172.21.15.12:0/258820032 learned_addr learned my addr 172.21.15.12:0/258820032 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f8666745640 1 --2- 172.21.15.12:0/258820032 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 0x7f86600d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f8665f44640 1 -- 172.21.15.12:0/258820032 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 msgr2=0x7f86600d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f8665f44640 1 --2- 172.21.15.12:0/258820032 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 0x7f86600d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f8665f44640 1 -- 172.21.15.12:0/258820032 >> v1:172.21.15.55:6789/0 conn(0x7f86600cfd20 legacy=0x7f86600d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:24.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.495+0000 7f8665f44640 1 -- 172.21.15.12:0/258820032 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600d2a40 con 0x7f866015e790 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.496+0000 7f8665f44640 1 --2- 172.21.15.12:0/258820032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f866015e790 0x7f86600d3240 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f86600d2f30 tx=0x7f865c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab2bcb78e3ecd5b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.496+0000 7f8664f42640 1 -- 172.21.15.12:0/258820032 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c035020 con 0x7f866015e790 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.496+0000 7f8664f42640 1 -- 172.21.15.12:0/258820032 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f865c031640 con 0x7f866015e790 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.496+0000 7f8664f42640 1 -- 172.21.15.12:0/258820032 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c02d040 con 0x7f866015e790 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.497+0000 7f86681a8640 1 -- 172.21.15.12:0/258820032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f866015e790 msgr2=0x7f86600d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.497+0000 7f86681a8640 1 --2- 172.21.15.12:0/258820032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f866015e790 0x7f86600d3240 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f86600d2f30 tx=0x7f865c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.497+0000 7f86681a8640 1 -- 172.21.15.12:0/258820032 shutdown_connections 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.497+0000 7f86681a8640 1 --2- 172.21.15.12:0/258820032 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 0x7f86600d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.497+0000 7f86681a8640 1 --2- 172.21.15.12:0/258820032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f866015e790 0x7f86600d3240 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.497+0000 7f86681a8640 1 -- 172.21.15.12:0/258820032 >> 172.21.15.12:0/258820032 conn(0x7f866008bf60 msgr2=0x7f86600d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.497+0000 7f86681a8640 1 -- 172.21.15.12:0/258820032 shutdown_connections 2024-04-15T15:23:24.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.497+0000 7f86681a8640 1 -- 172.21.15.12:0/258820032 wait complete. 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.498+0000 7f86681a8640 1 Processor -- start 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.498+0000 7f86681a8640 1 -- start start 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f86681a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86600cfd20 0x7f86601f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f86681a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 0x7f86600b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f86681a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f866015e790 0x7f8660150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f86681a8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f866015ecf0 con 0x7f86600cfd20 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f86681a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f866015eb70 con 0x7f866015e790 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f86681a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f86600b74f0 con 0x7f86600d1610 2024-04-15T15:23:24.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86600cfd20 0x7f86601f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86600cfd20 0x7f86601f54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46012/0 (socket says 172.21.15.12:46012) 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 -- 172.21.15.12:0/2379190398 learned_addr learned my addr 172.21.15.12:0/2379190398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8666745640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f866015e790 0x7f8660150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665743640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 0x7f86600b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 -- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 msgr2=0x7f86600b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 0x7f86600b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 -- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f866015e790 msgr2=0x7f8660150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f866015e790 0x7f8660150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 -- 172.21.15.12:0/2379190398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600b2430 con 0x7f86600cfd20 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8666745640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f866015e790 0x7f8660150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f8665f44640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86600cfd20 0x7f86601f54e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f865c000c00 tx=0x7f865c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.499+0000 7f86567fc640 1 -- 172.21.15.12:0/2379190398 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c040070 con 0x7f86600cfd20 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.500+0000 7f86567fc640 1 -- 172.21.15.12:0/2379190398 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f865c039c10 con 0x7f86600cfd20 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.500+0000 7f86567fc640 1 -- 172.21.15.12:0/2379190398 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c035040 con 0x7f86600cfd20 2024-04-15T15:23:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.500+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86600b3b40 con 0x7f86600cfd20 2024-04-15T15:23:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.500+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86600b4100 con 0x7f86600cfd20 2024-04-15T15:23:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.501+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86600689e0 con 0x7f86600cfd20 2024-04-15T15:23:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.501+0000 7f86567fc640 1 -- 172.21.15.12:0/2379190398 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f865c04b060 con 0x7f86600cfd20 2024-04-15T15:23:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.502+0000 7f86567fc640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f864406c9b0 0x7f864406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.502+0000 7f86567fc640 1 -- 172.21.15.12:0/2379190398 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f865c0ad740 con 0x7f86600cfd20 2024-04-15T15:23:24.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.502+0000 7f8665743640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f864406c9b0 0x7f864406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.502+0000 7f8665743640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f864406c9b0 0x7f864406ee70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f86600b6030 tx=0x7f8650009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.506+0000 7f86567fc640 1 -- 172.21.15.12:0/2379190398 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f865c031610 con 0x7f86600cfd20 2024-04-15T15:23:24.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.692+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86600d1ce0 con 0x7f86600cfd20 2024-04-15T15:23:24.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.693+0000 7f86567fc640 1 -- 172.21.15.12:0/2379190398 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17084 (secure 0 0 0) 0x7f865c07ba10 con 0x7f86600cfd20 2024-04-15T15:23:24.696 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:24.696 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":42,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:23:15.001862+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6837/3388147599":"2024-04-16T15:23:15.001839+0000","172.21.15.55:6835/3388147599":"2024-04-16T15:23:15.001839+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:23:24.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f864406c9b0 msgr2=0x7f864406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f864406c9b0 0x7f864406ee70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f86600b6030 tx=0x7f8650009040 comp rx=0 tx=0).stop 2024-04-15T15:23:24.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86600cfd20 msgr2=0x7f86601f54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86600cfd20 0x7f86601f54e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f865c000c00 tx=0x7f865c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 shutdown_connections 2024-04-15T15:23:24.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f864406c9b0 0x7f864406ee70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f866015e790 0x7f8660150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f86600d1610 0x7f86600b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 --2- 172.21.15.12:0/2379190398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86600cfd20 0x7f86601f54e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.695+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 >> 172.21.15.12:0/2379190398 conn(0x7f866008bf60 msgr2=0x7f86600b9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.696+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 shutdown_connections 2024-04-15T15:23:24.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.696+0000 7f86681a8640 1 -- 172.21.15.12:0/2379190398 wait complete. 2024-04-15T15:23:24.709 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:23:24.710 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:23:24.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.815+0000 7f4f90340640 1 Processor -- start 2024-04-15T15:23:24.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.815+0000 7f4f90340640 1 -- start start 2024-04-15T15:23:24.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f90340640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f8815d090 0x7f4f88160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f90340640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4f88067720 0x7f4f8806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f90340640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f8815d660 con 0x7f4f8806ea30 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f90340640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4f88160130 con 0x7f4f8815d090 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f90340640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4f881610c0 con 0x7f4f88067720 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8e0dc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f8815d090 0x7f4f88160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8e8dd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4f8806ea30 0x7f4f880cd550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47032/0 (socket says 172.21.15.12:47032) 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8e0dc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f8815d090 0x7f4f88160b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46016/0 (socket says 172.21.15.12:46016) 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8e8dd640 1 -- 172.21.15.12:0/1437240596 learned_addr learned my addr 172.21.15.12:0/1437240596 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8d8db640 1 --2- 172.21.15.12:0/1437240596 >> v2:172.21.15.55:3300/0 conn(0x7f4f88067720 0x7f4f8806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8e0dc640 1 -- 172.21.15.12:0/1437240596 >> v2:172.21.15.55:3300/0 conn(0x7f4f88067720 msgr2=0x7f4f8806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8e0dc640 1 --2- 172.21.15.12:0/1437240596 >> v2:172.21.15.55:3300/0 conn(0x7f4f88067720 0x7f4f8806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8e0dc640 1 -- 172.21.15.12:0/1437240596 >> v1:172.21.15.12:6789/0 conn(0x7f4f8806ea30 legacy=0x7f4f880cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8d0da640 1 -- 172.21.15.12:0/1437240596 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587641056 0 0) 0x7f4f8815d660 con 0x7f4f8806ea30 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8e0dc640 1 -- 172.21.15.12:0/1437240596 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8815d940 con 0x7f4f8815d090 2024-04-15T15:23:24.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.816+0000 7f4f8d8db640 1 --2- 172.21.15.12:0/1437240596 >> v2:172.21.15.55:3300/0 conn(0x7f4f88067720 0x7f4f8806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:24.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.817+0000 7f4f8e0dc640 1 --2- 172.21.15.12:0/1437240596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8815d090 0x7f4f88160b80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4f880b6240 tx=0x7f4f8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c722e713ec596489 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:24.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.817+0000 7f4f8d0da640 1 -- 172.21.15.12:0/1437240596 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f840315a0 con 0x7f4f8815d090 2024-04-15T15:23:24.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.817+0000 7f4f8d0da640 1 -- 172.21.15.12:0/1437240596 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f84037070 con 0x7f4f8815d090 2024-04-15T15:23:24.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.817+0000 7f4f8d0da640 1 -- 172.21.15.12:0/1437240596 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f84031960 con 0x7f4f8815d090 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.818+0000 7f4f90340640 1 -- 172.21.15.12:0/1437240596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8815d090 msgr2=0x7f4f88160b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.818+0000 7f4f90340640 1 --2- 172.21.15.12:0/1437240596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8815d090 0x7f4f88160b80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4f880b6240 tx=0x7f4f8402b840 comp rx=0 tx=0).stop 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.818+0000 7f4f90340640 1 -- 172.21.15.12:0/1437240596 shutdown_connections 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.818+0000 7f4f90340640 1 --2- 172.21.15.12:0/1437240596 >> v2:172.21.15.55:3300/0 conn(0x7f4f88067720 0x7f4f8806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.818+0000 7f4f90340640 1 --2- 172.21.15.12:0/1437240596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8815d090 0x7f4f88160b80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.818+0000 7f4f90340640 1 -- 172.21.15.12:0/1437240596 >> 172.21.15.12:0/1437240596 conn(0x7f4f88073360 msgr2=0x7f4f8806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.818+0000 7f4f90340640 1 -- 172.21.15.12:0/1437240596 shutdown_connections 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.818+0000 7f4f90340640 1 -- 172.21.15.12:0/1437240596 wait complete. 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.819+0000 7f4f90340640 1 Processor -- start 2024-04-15T15:23:24.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.819+0000 7f4f90340640 1 -- start start 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f90340640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f88067720 0x7f4f88152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f90340640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f8806ea30 0x7f4f8815f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f90340640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f8815d090 0x7f4f8815ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f90340640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4f880b0460 con 0x7f4f88067720 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f90340640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4f88160660 con 0x7f4f8806ea30 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f90340640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f4f880b05e0 con 0x7f4f8815d090 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f88067720 0x7f4f88152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f88067720 0x7f4f88152e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46022/0 (socket says 172.21.15.12:46022) 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 -- 172.21.15.12:0/4224315040 learned_addr learned my addr 172.21.15.12:0/4224315040 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8d8db640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f8806ea30 0x7f4f8815f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e8dd640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f8815d090 0x7f4f8815ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 -- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f8815d090 msgr2=0x7f4f8815ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f8815d090 0x7f4f8815ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 -- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f8806ea30 msgr2=0x7f4f8815f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f8806ea30 0x7f4f8815f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 -- 172.21.15.12:0/4224315040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8815c660 con 0x7f4f88067720 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f8e0dc640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f88067720 0x7f4f88152e40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4f84000c00 tx=0x7f4f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.820+0000 7f4f7effd640 1 -- 172.21.15.12:0/4224315040 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f84004230 con 0x7f4f88067720 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.821+0000 7f4f7effd640 1 -- 172.21.15.12:0/4224315040 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f840043d0 con 0x7f4f88067720 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.821+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f881f2e30 con 0x7f4f88067720 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.821+0000 7f4f7effd640 1 -- 172.21.15.12:0/4224315040 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f84003850 con 0x7f4f88067720 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.821+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f881f2b50 con 0x7f4f88067720 2024-04-15T15:23:24.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.822+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f880689e0 con 0x7f4f88067720 2024-04-15T15:23:24.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.822+0000 7f4f7effd640 1 -- 172.21.15.12:0/4224315040 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4f840039f0 con 0x7f4f88067720 2024-04-15T15:23:24.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.822+0000 7f4f7effd640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f6c06c980 0x7f4f6c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:24.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.822+0000 7f4f7effd640 1 -- 172.21.15.12:0/4224315040 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f4f8402f030 con 0x7f4f88067720 2024-04-15T15:23:24.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.823+0000 7f4f8d8db640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f6c06c980 0x7f4f6c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:24.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.823+0000 7f4f8d8db640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f6c06c980 0x7f4f6c06ee40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4f88154220 tx=0x7f4f78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:24.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:24.827+0000 7f4f7effd640 1 -- 172.21.15.12:0/4224315040 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f84041070 con 0x7f4f88067720 2024-04-15T15:23:25.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.009+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f88160cd0 con 0x7f4f88067720 2024-04-15T15:23:25.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.010+0000 7f4f7effd640 1 -- 172.21.15.12:0/4224315040 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f4f84003ce0 con 0x7f4f88067720 2024-04-15T15:23:25.012 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:23:25.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f6c06c980 msgr2=0x7f4f6c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f6c06c980 0x7f4f6c06ee40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4f88154220 tx=0x7f4f78008040 comp rx=0 tx=0).stop 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f88067720 msgr2=0x7f4f88152e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f88067720 0x7f4f88152e40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4f84000c00 tx=0x7f4f840046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 shutdown_connections 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f8815d090 0x7f4f8815ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f6c06c980 0x7f4f6c06ee40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f8806ea30 0x7f4f8815f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 --2- 172.21.15.12:0/4224315040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f88067720 0x7f4f88152e40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 >> 172.21.15.12:0/4224315040 conn(0x7f4f88073360 msgr2=0x7f4f880b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 shutdown_connections 2024-04-15T15:23:25.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.012+0000 7f4f90340640 1 -- 172.21.15.12:0/4224315040 wait complete. 2024-04-15T15:23:25.025 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:23:25.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.131+0000 7fca43d32640 1 Processor -- start 2024-04-15T15:23:25.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.131+0000 7fca43d32640 1 -- start start 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca43d32640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fca3c15e790 0x7fca3c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca43d32640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 0x7fca3c0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca43d32640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3c0d2480 con 0x7fca3c0d0f00 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca43d32640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fca3c0d2db0 con 0x7fca3c15e790 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca43d32640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c0d2760 con 0x7fca3c0cfd20 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca422cf640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fca3c0d0f00 0x7fca3c0b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47042/0 (socket says 172.21.15.12:47042) 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca422cf640 1 -- 172.21.15.12:0/3385626064 learned_addr learned my addr 172.21.15.12:0/3385626064 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca41ace640 1 --2- 172.21.15.12:0/3385626064 >> v2:172.21.15.55:3300/0 conn(0x7fca3c15e790 0x7fca3c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca412cd640 1 --2- 172.21.15.12:0/3385626064 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 0x7fca3c0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca40acc640 1 -- 172.21.15.12:0/3385626064 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3364813833 0 0) 0x7fca3c0d2480 con 0x7fca3c0d0f00 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca40acc640 1 -- 172.21.15.12:0/3385626064 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca28003520 con 0x7fca3c0d0f00 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca41ace640 1 -- 172.21.15.12:0/3385626064 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 msgr2=0x7fca3c0d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:25.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca41ace640 1 --2- 172.21.15.12:0/3385626064 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 0x7fca3c0d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca41ace640 1 -- 172.21.15.12:0/3385626064 >> v1:172.21.15.12:6789/0 conn(0x7fca3c0d0f00 legacy=0x7fca3c0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca40acc640 1 -- 172.21.15.12:0/3385626064 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 704361870 0 0) 0x7fca28003520 con 0x7fca3c0d0f00 2024-04-15T15:23:25.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.132+0000 7fca41ace640 1 -- 172.21.15.12:0/3385626064 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c0d2a40 con 0x7fca3c15e790 2024-04-15T15:23:25.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.133+0000 7fca41ace640 1 --2- 172.21.15.12:0/3385626064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 0x7fca3c0d3240 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fca380097a0 tx=0x7fca3802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4442ed1efab0b494 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca40acc640 1 -- 172.21.15.12:0/3385626064 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38037070 con 0x7fca3c15e790 2024-04-15T15:23:25.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca40acc640 1 -- 172.21.15.12:0/3385626064 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca38030510 con 0x7fca3c15e790 2024-04-15T15:23:25.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca40acc640 1 -- 172.21.15.12:0/3385626064 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca340019f0 con 0x7fca3c15e790 2024-04-15T15:23:25.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca43d32640 1 -- 172.21.15.12:0/3385626064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 msgr2=0x7fca3c0d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca43d32640 1 --2- 172.21.15.12:0/3385626064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 0x7fca3c0d3240 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fca380097a0 tx=0x7fca3802b790 comp rx=0 tx=0).stop 2024-04-15T15:23:25.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca43d32640 1 -- 172.21.15.12:0/3385626064 shutdown_connections 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca43d32640 1 --2- 172.21.15.12:0/3385626064 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 0x7fca3c0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca43d32640 1 --2- 172.21.15.12:0/3385626064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 0x7fca3c0d3240 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca43d32640 1 -- 172.21.15.12:0/3385626064 >> 172.21.15.12:0/3385626064 conn(0x7fca3c08bf60 msgr2=0x7fca3c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca43d32640 1 -- 172.21.15.12:0/3385626064 shutdown_connections 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.134+0000 7fca43d32640 1 -- 172.21.15.12:0/3385626064 wait complete. 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.135+0000 7fca43d32640 1 Processor -- start 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.135+0000 7fca43d32640 1 -- start start 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca43d32640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 0x7fca3c151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca43d32640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca3c0d0f00 0x7fca3c152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca43d32640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 0x7fca3c0c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca43d32640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c15ecf0 con 0x7fca3c0d0f00 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca43d32640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c15eb70 con 0x7fca3c15e790 2024-04-15T15:23:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca43d32640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c0d13b0 con 0x7fca3c0cfd20 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca412cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca3c0d0f00 0x7fca3c152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca412cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca3c0d0f00 0x7fca3c152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46032/0 (socket says 172.21.15.12:46032) 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.136+0000 7fca412cd640 1 -- 172.21.15.12:0/922589719 learned_addr learned my addr 172.21.15.12:0/922589719 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca422cf640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 0x7fca3c0c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca41ace640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 0x7fca3c151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca412cd640 1 -- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 msgr2=0x7fca3c151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca412cd640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 0x7fca3c151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca412cd640 1 -- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 msgr2=0x7fca3c0c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca412cd640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 0x7fca3c0c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca412cd640 1 -- 172.21.15.12:0/922589719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c09fb10 con 0x7fca3c0d0f00 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca412cd640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca3c0d0f00 0x7fca3c152660 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00e930 tx=0x7fca2c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca327fc640 1 -- 172.21.15.12:0/922589719 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c017070 con 0x7fca3c0d0f00 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca327fc640 1 -- 172.21.15.12:0/922589719 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2c00ca00 con 0x7fca3c0d0f00 2024-04-15T15:23:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3c153120 con 0x7fca3c0d0f00 2024-04-15T15:23:25.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca327fc640 1 -- 172.21.15.12:0/922589719 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c00cdf0 con 0x7fca3c0d0f00 2024-04-15T15:23:25.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.137+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3c1539c0 con 0x7fca3c0d0f00 2024-04-15T15:23:25.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.138+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca3c153510 con 0x7fca3c0d0f00 2024-04-15T15:23:25.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.138+0000 7fca327fc640 1 -- 172.21.15.12:0/922589719 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fca2c01a070 con 0x7fca3c0d0f00 2024-04-15T15:23:25.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.139+0000 7fca327fc640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fca2006c980 0x7fca2006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.139+0000 7fca327fc640 1 -- 172.21.15.12:0/922589719 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fca2c08bdf0 con 0x7fca3c0d0f00 2024-04-15T15:23:25.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.139+0000 7fca41ace640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fca2006c980 0x7fca2006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.139+0000 7fca41ace640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fca2006c980 0x7fca2006ee40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fca38034000 tx=0x7fca3802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.143+0000 7fca327fc640 1 -- 172.21.15.12:0/922589719 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca2c05a0f0 con 0x7fca3c0d0f00 2024-04-15T15:23:25.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.327+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fca3c0689e0 con 0x7fca3c0d0f00 2024-04-15T15:23:25.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.327+0000 7fca327fc640 1 -- 172.21.15.12:0/922589719 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7fca2c0599f0 con 0x7fca3c0d0f00 2024-04-15T15:23:25.330 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fca2006c980 msgr2=0x7fca2006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fca2006c980 0x7fca2006ee40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fca38034000 tx=0x7fca3802f000 comp rx=0 tx=0).stop 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca3c0d0f00 msgr2=0x7fca3c152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca3c0d0f00 0x7fca3c152660 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00e930 tx=0x7fca2c00edf0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 shutdown_connections 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fca2006c980 0x7fca2006ee40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca3c15e790 0x7fca3c0c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca3c0d0f00 0x7fca3c152660 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 --2- 172.21.15.12:0/922589719 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fca3c0cfd20 0x7fca3c151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 >> 172.21.15.12:0/922589719 conn(0x7fca3c08bf60 msgr2=0x7fca3c06cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 shutdown_connections 2024-04-15T15:23:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.329+0000 7fca43d32640 1 -- 172.21.15.12:0/922589719 wait complete. 2024-04-15T15:23:25.342 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:23:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.447+0000 7f2ca7435640 1 Processor -- start 2024-04-15T15:23:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.447+0000 7f2ca7435640 1 -- start start 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca7435640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca015e790 0x7f2ca00b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca7435640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ca00b7be0 0x7f2ca00d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca7435640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ca015eb70 con 0x7f2ca00d1b20 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca7435640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2ca00d3790 con 0x7f2ca00b7be0 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca7435640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca00d45b0 con 0x7f2ca015e790 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca49d0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ca00b7be0 0x7f2ca00d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca49d0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ca00b7be0 0x7f2ca00d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46038/0 (socket says 172.21.15.12:46038) 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca49d0640 1 -- 172.21.15.12:0/4214574757 learned_addr learned my addr 172.21.15.12:0/4214574757 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca51d1640 1 --2- 172.21.15.12:0/4214574757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca015e790 0x7f2ca00b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca49d0640 1 -- 172.21.15.12:0/4214574757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca015e790 msgr2=0x7f2ca00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca49d0640 1 --2- 172.21.15.12:0/4214574757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca015e790 0x7f2ca00b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca49d0640 1 -- 172.21.15.12:0/4214574757 >> v1:172.21.15.55:6789/0 conn(0x7f2ca00d1b20 legacy=0x7f2ca00d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.448+0000 7f2ca49d0640 1 -- 172.21.15.12:0/4214574757 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca00d3c80 con 0x7f2ca00b7be0 2024-04-15T15:23:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.449+0000 7f2ca49d0640 1 --2- 172.21.15.12:0/4214574757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00d15e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009870 tx=0x7f2c9002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9fca00609162e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.449+0000 7f2c97fff640 1 -- 172.21.15.12:0/4214574757 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c900317c0 con 0x7f2ca00b7be0 2024-04-15T15:23:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.449+0000 7f2c97fff640 1 -- 172.21.15.12:0/4214574757 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c90037070 con 0x7f2ca00b7be0 2024-04-15T15:23:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.449+0000 7f2c97fff640 1 -- 172.21.15.12:0/4214574757 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c90031b80 con 0x7f2ca00b7be0 2024-04-15T15:23:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.450+0000 7f2ca7435640 1 -- 172.21.15.12:0/4214574757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 msgr2=0x7f2ca00d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.450+0000 7f2ca7435640 1 --2- 172.21.15.12:0/4214574757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00d15e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009870 tx=0x7f2c9002b860 comp rx=0 tx=0).stop 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.450+0000 7f2ca7435640 1 -- 172.21.15.12:0/4214574757 shutdown_connections 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.450+0000 7f2ca7435640 1 --2- 172.21.15.12:0/4214574757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00d15e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.450+0000 7f2ca7435640 1 --2- 172.21.15.12:0/4214574757 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca015e790 0x7f2ca00b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.450+0000 7f2ca7435640 1 -- 172.21.15.12:0/4214574757 >> 172.21.15.12:0/4214574757 conn(0x7f2ca0067b40 msgr2=0x7f2ca0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.450+0000 7f2ca7435640 1 -- 172.21.15.12:0/4214574757 shutdown_connections 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.450+0000 7f2ca7435640 1 -- 172.21.15.12:0/4214574757 wait complete. 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.451+0000 7f2ca7435640 1 Processor -- start 2024-04-15T15:23:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.451+0000 7f2ca7435640 1 -- start start 2024-04-15T15:23:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca7435640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca7435640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca00d1b20 0x7f2ca01f80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca7435640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca015e790 0x7f2ca01e9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca7435640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca00d5150 con 0x7f2ca00b7be0 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca7435640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca00d4240 con 0x7f2ca015e790 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca7435640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca00d52d0 con 0x7f2ca00d1b20 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46052/0 (socket says 172.21.15.12:46052) 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 -- 172.21.15.12:0/334186224 learned_addr learned my addr 172.21.15.12:0/334186224 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca49d0640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca00d1b20 0x7f2ca01f80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca59d2640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca015e790 0x7f2ca01e9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 -- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca00d1b20 msgr2=0x7f2ca01f80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca00d1b20 0x7f2ca01f80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 -- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca015e790 msgr2=0x7f2ca01e9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca015e790 0x7f2ca01e9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 -- 172.21.15.12:0/334186224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca00c9ae0 con 0x7f2ca00b7be0 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2ca51d1640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00c9170 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c00e9b0 tx=0x7f2c9c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.452+0000 7f2c95ffb640 1 -- 172.21.15.12:0/334186224 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c017070 con 0x7f2ca00b7be0 2024-04-15T15:23:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.453+0000 7f2c95ffb640 1 -- 172.21.15.12:0/334186224 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c9c00c9a0 con 0x7f2ca00b7be0 2024-04-15T15:23:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.453+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca00c9dc0 con 0x7f2ca00b7be0 2024-04-15T15:23:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.453+0000 7f2c95ffb640 1 -- 172.21.15.12:0/334186224 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c00cd90 con 0x7f2ca00b7be0 2024-04-15T15:23:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.453+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca00ca380 con 0x7f2ca00b7be0 2024-04-15T15:23:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.454+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca00689e0 con 0x7f2ca00b7be0 2024-04-15T15:23:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.454+0000 7f2c95ffb640 1 -- 172.21.15.12:0/334186224 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2c9c01a070 con 0x7f2ca00b7be0 2024-04-15T15:23:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.455+0000 7f2c95ffb640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c8406c980 0x7f2c8406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.455+0000 7f2c95ffb640 1 -- 172.21.15.12:0/334186224 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f2c9c08c070 con 0x7f2ca00b7be0 2024-04-15T15:23:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.455+0000 7f2ca49d0640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c8406c980 0x7f2c8406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.455+0000 7f2ca49d0640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c8406c980 0x7f2c8406ee40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2c90036000 tx=0x7f2c9003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.459+0000 7f2c95ffb640 1 -- 172.21.15.12:0/334186224 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c9c059c70 con 0x7f2ca00b7be0 2024-04-15T15:23:25.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.643+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2ca00ce4f0 con 0x7f2ca00b7be0 2024-04-15T15:23:25.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.643+0000 7f2c95ffb640 1 -- 172.21.15.12:0/334186224 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f2c9c05df70 con 0x7f2ca00b7be0 2024-04-15T15:23:25.646 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c8406c980 msgr2=0x7f2c8406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c8406c980 0x7f2c8406ee40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2c90036000 tx=0x7f2c9003d040 comp rx=0 tx=0).stop 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 msgr2=0x7f2ca00c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00c9170 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c00e9b0 tx=0x7f2c9c00ee70 comp rx=0 tx=0).stop 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 shutdown_connections 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f2c8406c980 0x7f2c8406ee40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ca015e790 0x7f2ca01e9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f2ca00d1b20 0x7f2ca01f80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 --2- 172.21.15.12:0/334186224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ca00b7be0 0x7f2ca00c9170 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 >> 172.21.15.12:0/334186224 conn(0x7f2ca0067b40 msgr2=0x7f2ca00cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 shutdown_connections 2024-04-15T15:23:25.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.645+0000 7f2ca7435640 1 -- 172.21.15.12:0/334186224 wait complete. 2024-04-15T15:23:25.659 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:23:25.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.762+0000 7f9fe2540640 1 Processor -- start 2024-04-15T15:23:25.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.763+0000 7f9fe2540640 1 -- start start 2024-04-15T15:23:25.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.763+0000 7f9fe2540640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fdc0d2360 0x7f9fdc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.763+0000 7f9fe2540640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fdc0b47a0 con 0x7f9fdc15eae0 2024-04-15T15:23:25.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.763+0000 7f9fe2540640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fdc0b4a80 con 0x7f9fdc15e700 2024-04-15T15:23:25.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.763+0000 7f9fe2540640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9fdc0b53b0 con 0x7f9fdc0d2360 2024-04-15T15:23:25.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.764+0000 7f9fd3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fdc0d2360 0x7f9fdc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.764+0000 7f9fe0add640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9fdc15eae0 0x7f9fdc0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47056/0 (socket says 172.21.15.12:47056) 2024-04-15T15:23:25.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.764+0000 7f9fe0add640 1 -- 172.21.15.12:0/1424968005 learned_addr learned my addr 172.21.15.12:0/1424968005 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:25.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.764+0000 7f9fd3fff640 1 -- 172.21.15.12:0/1424968005 >> v1:172.21.15.55:6789/0 conn(0x7f9fdc15e700 legacy=0x7f9fdc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.764+0000 7f9fd3fff640 1 -- 172.21.15.12:0/1424968005 >> v1:172.21.15.12:6789/0 conn(0x7f9fdc15eae0 legacy=0x7f9fdc0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.764+0000 7f9fdb7fe640 1 -- 172.21.15.12:0/1424968005 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1998596045 0 0) 0x7f9fdc0b47a0 con 0x7f9fdc15eae0 2024-04-15T15:23:25.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.764+0000 7f9fd3fff640 1 -- 172.21.15.12:0/1424968005 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc0b4d60 con 0x7f9fdc0d2360 2024-04-15T15:23:25.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fd3fff640 1 --2- 172.21.15.12:0/1424968005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc0d2360 0x7f9fdc0d2b60 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0b5530 tx=0x7f9fc402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0e2b3fa83d5e9a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fdb7fe640 1 -- 172.21.15.12:0/1424968005 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4034070 con 0x7f9fdc0d2360 2024-04-15T15:23:25.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fdb7fe640 1 -- 172.21.15.12:0/1424968005 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc4030610 con 0x7f9fdc0d2360 2024-04-15T15:23:25.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fdb7fe640 1 -- 172.21.15.12:0/1424968005 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc40309e0 con 0x7f9fdc0d2360 2024-04-15T15:23:25.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fe2540640 1 -- 172.21.15.12:0/1424968005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc0d2360 msgr2=0x7f9fdc0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fe2540640 1 --2- 172.21.15.12:0/1424968005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc0d2360 0x7f9fdc0d2b60 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0b5530 tx=0x7f9fc402b960 comp rx=0 tx=0).stop 2024-04-15T15:23:25.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fe2540640 1 -- 172.21.15.12:0/1424968005 shutdown_connections 2024-04-15T15:23:25.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fe2540640 1 --2- 172.21.15.12:0/1424968005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc0d2360 0x7f9fdc0d2b60 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fe2540640 1 -- 172.21.15.12:0/1424968005 >> 172.21.15.12:0/1424968005 conn(0x7f9fdc08bf60 msgr2=0x7f9fdc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:25.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fe2540640 1 -- 172.21.15.12:0/1424968005 shutdown_connections 2024-04-15T15:23:25.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.765+0000 7f9fe2540640 1 -- 172.21.15.12:0/1424968005 wait complete. 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fe2540640 1 Processor -- start 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fe2540640 1 -- start start 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fe2540640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fdc0d2360 0x7f9fdc0c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fe2540640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc15e700 0x7f9fdc0cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fe2540640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fdc15eae0 0x7f9fdc1f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fe2540640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc0d1d40 con 0x7f9fdc15e700 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fe2540640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc0b4450 con 0x7f9fdc15eae0 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fe2540640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9fdc0d1ec0 con 0x7f9fdc0d2360 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fd3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc15e700 0x7f9fdc0cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fd3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc15e700 0x7f9fdc0cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46072/0 (socket says 172.21.15.12:46072) 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.767+0000 7f9fd3fff640 1 -- 172.21.15.12:0/2580328061 learned_addr learned my addr 172.21.15.12:0/2580328061 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fe0add640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fdc15eae0 0x7f9fdc1f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd3fff640 1 -- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fdc0d2360 msgr2=0x7f9fdc0c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fdbfff640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fdc0d2360 0x7f9fdc0c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd3fff640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fdc0d2360 0x7f9fdc0c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd3fff640 1 -- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fdc15eae0 msgr2=0x7f9fdc1f7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd3fff640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fdc15eae0 0x7f9fdc1f7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd3fff640 1 -- 172.21.15.12:0/2580328061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc0ce120 con 0x7f9fdc15e700 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fe0add640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fdc15eae0 0x7f9fdc1f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd3fff640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc15e700 0x7f9fdc0cb650 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4000c00 tx=0x7f9fc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd97fa640 1 -- 172.21.15.12:0/2580328061 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4034040 con 0x7f9fdc15e700 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd97fa640 1 -- 172.21.15.12:0/2580328061 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc4004030 con 0x7f9fdc15e700 2024-04-15T15:23:25.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fdc0ce400 con 0x7f9fdc15e700 2024-04-15T15:23:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fd97fa640 1 -- 172.21.15.12:0/2580328061 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4004420 con 0x7f9fdc15e700 2024-04-15T15:23:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.768+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fdc0cae30 con 0x7f9fdc15e700 2024-04-15T15:23:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.769+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fdc0689e0 con 0x7f9fdc15e700 2024-04-15T15:23:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.769+0000 7f9fd97fa640 1 -- 172.21.15.12:0/2580328061 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9fc4037070 con 0x7f9fdc15e700 2024-04-15T15:23:25.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.770+0000 7f9fd97fa640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fbc06c980 0x7f9fbc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:25.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.770+0000 7f9fd97fa640 1 -- 172.21.15.12:0/2580328061 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f9fc40add70 con 0x7f9fdc15e700 2024-04-15T15:23:25.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.770+0000 7f9fdbfff640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fbc06c980 0x7f9fbc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:25.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.771+0000 7f9fdbfff640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fbc06c980 0x7f9fbc06ee40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc000fb0 tx=0x7f9fcc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:25.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.775+0000 7f9fd97fa640 1 -- 172.21.15.12:0/2580328061 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fc407bfc0 con 0x7f9fdc15e700 2024-04-15T15:23:25.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.970+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9fdc0ba500 con 0x7f9fdc15e700 2024-04-15T15:23:25.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.970+0000 7f9fd97fa640 1 -- 172.21.15.12:0/2580328061 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7f9fc407b8c0 con 0x7f9fdc15e700 2024-04-15T15:23:25.973 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.55:6837/3388147599 2024-04-16T15:23:15.001839+0000 2024-04-15T15:23:25.973 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.55:6835/3388147599 2024-04-16T15:23:15.001839+0000 2024-04-15T15:23:25.973 INFO:teuthology.orchestra.run.smithi012.stderr:listed 2 entries 2024-04-15T15:23:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fbc06c980 msgr2=0x7f9fbc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fbc06c980 0x7f9fbc06ee40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc000fb0 tx=0x7f9fcc009040 comp rx=0 tx=0).stop 2024-04-15T15:23:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc15e700 msgr2=0x7f9fdc0cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc15e700 0x7f9fdc0cb650 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4000c00 tx=0x7f9fc40046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 shutdown_connections 2024-04-15T15:23:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9fbc06c980 0x7f9fbc06ee40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fdc15eae0 0x7f9fdc1f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fdc15e700 0x7f9fdc0cb650 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 --2- 172.21.15.12:0/2580328061 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9fdc0d2360 0x7f9fdc0c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 >> 172.21.15.12:0/2580328061 conn(0x7f9fdc08bf60 msgr2=0x7f9fdc0b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 shutdown_connections 2024-04-15T15:23:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:25.972+0000 7f9fe2540640 1 -- 172.21.15.12:0/2580328061 wait complete. 2024-04-15T15:23:25.986 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-04-15T15:23:25.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:25.986 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-15T15:23:26.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi012.stderr:ceph-fuse[56052]: fuse finished with error 0 and tester_r 0 2024-04-15T15:23:26.014 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:23:26.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:26.015 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:23:26.182 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:23:26.183 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:23:26.183 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:26.203 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:23:26.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.326+0000 7f943b863640 1 Processor -- start 2024-04-15T15:23:26.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.326+0000 7f943b863640 1 -- start start 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.327+0000 7f943b863640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f943409b0e0 0x7f9434070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.327+0000 7f943b863640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94340cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.327+0000 7f943b863640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f943406ff30 con 0x7f94340ce8e0 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.327+0000 7f943b863640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9434070820 con 0x7f943409b0e0 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.327+0000 7f943b863640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f94340cecc0 con 0x7f9434071d90 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.327+0000 7f9438dfe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f943409b0e0 0x7f9434070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.327+0000 7f9439e00640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94340cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.328+0000 7f94395ff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f94340ce8e0 0x7f943409a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:42744/0 (socket says 172.21.15.12:42744) 2024-04-15T15:23:26.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.328+0000 7f94395ff640 1 -- 172.21.15.12:0/3556866127 learned_addr learned my addr 172.21.15.12:0/3556866127 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.328+0000 7f9439e00640 1 -- 172.21.15.12:0/3556866127 >> v2:172.21.15.55:3300/0 conn(0x7f943409b0e0 msgr2=0x7f9434070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.328+0000 7f9439e00640 1 --2- 172.21.15.12:0/3556866127 >> v2:172.21.15.55:3300/0 conn(0x7f943409b0e0 0x7f9434070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.328+0000 7f9439e00640 1 -- 172.21.15.12:0/3556866127 >> v1:172.21.15.55:6789/0 conn(0x7f94340ce8e0 legacy=0x7f943409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.328+0000 7f9439e00640 1 -- 172.21.15.12:0/3556866127 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943409b9d0 con 0x7f9434071d90 2024-04-15T15:23:26.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.329+0000 7f9439e00640 1 --2- 172.21.15.12:0/3556866127 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94340cce80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f942c009d90 tx=0x7f942c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=991089f33561e370 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.329+0000 7f942bfff640 1 -- 172.21.15.12:0/3556866127 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c039070 con 0x7f9434071d90 2024-04-15T15:23:26.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.329+0000 7f942bfff640 1 -- 172.21.15.12:0/3556866127 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f942c02cbc0 con 0x7f9434071d90 2024-04-15T15:23:26.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.329+0000 7f942bfff640 1 -- 172.21.15.12:0/3556866127 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c034440 con 0x7f9434071d90 2024-04-15T15:23:26.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.330+0000 7f943b863640 1 -- 172.21.15.12:0/3556866127 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 msgr2=0x7f94340cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.330+0000 7f943b863640 1 --2- 172.21.15.12:0/3556866127 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94340cce80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f942c009d90 tx=0x7f942c02be10 comp rx=0 tx=0).stop 2024-04-15T15:23:26.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.330+0000 7f943b863640 1 -- 172.21.15.12:0/3556866127 shutdown_connections 2024-04-15T15:23:26.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.330+0000 7f943b863640 1 --2- 172.21.15.12:0/3556866127 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94340cce80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.330+0000 7f943b863640 1 --2- 172.21.15.12:0/3556866127 >> v2:172.21.15.55:3300/0 conn(0x7f943409b0e0 0x7f9434070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.330+0000 7f943b863640 1 -- 172.21.15.12:0/3556866127 >> 172.21.15.12:0/3556866127 conn(0x7f94340b3f40 msgr2=0x7f94340b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:26.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.330+0000 7f943b863640 1 -- 172.21.15.12:0/3556866127 shutdown_connections 2024-04-15T15:23:26.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.330+0000 7f943b863640 1 -- 172.21.15.12:0/3556866127 wait complete. 2024-04-15T15:23:26.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.331+0000 7f943b863640 1 Processor -- start 2024-04-15T15:23:26.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.331+0000 7f943b863640 1 -- start start 2024-04-15T15:23:26.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f943b863640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94341f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f943b863640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943409b0e0 0x7f94341e6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f943b863640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94340ce8e0 0x7f94341f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f943b863640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f94340704b0 con 0x7f94340ce8e0 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f943b863640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f94340701d0 con 0x7f943409b0e0 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f943b863640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9434071970 con 0x7f9434071d90 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94340ce8e0 0x7f94341f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94340ce8e0 0x7f94341f7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46084/0 (socket says 172.21.15.12:46084) 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 -- 172.21.15.12:0/1732482044 learned_addr learned my addr 172.21.15.12:0/1732482044 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9438dfe640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943409b0e0 0x7f94341e6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f94395ff640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94341f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 -- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 msgr2=0x7f94341f5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94341f5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 -- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943409b0e0 msgr2=0x7f94341e6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943409b0e0 0x7f94341e6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 -- 172.21.15.12:0/1732482044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94341e5150 con 0x7f94340ce8e0 2024-04-15T15:23:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.332+0000 7f9439e00640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94340ce8e0 0x7f94341f7520 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f942c009d90 tx=0x7f942c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.333+0000 7f9429ffb640 1 -- 172.21.15.12:0/1732482044 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c039050 con 0x7f94340ce8e0 2024-04-15T15:23:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.333+0000 7f9429ffb640 1 -- 172.21.15.12:0/1732482044 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f942c045070 con 0x7f94340ce8e0 2024-04-15T15:23:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.333+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f943406cd00 con 0x7f94340ce8e0 2024-04-15T15:23:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.333+0000 7f9429ffb640 1 -- 172.21.15.12:0/1732482044 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c009450 con 0x7f94340ce8e0 2024-04-15T15:23:26.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.333+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f943406d2c0 con 0x7f94340ce8e0 2024-04-15T15:23:26.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.334+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94340b1b20 con 0x7f94340ce8e0 2024-04-15T15:23:26.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.334+0000 7f9429ffb640 1 -- 172.21.15.12:0/1732482044 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f942c03e850 con 0x7f94340ce8e0 2024-04-15T15:23:26.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.335+0000 7f9429ffb640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f941806c980 0x7f941806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.335+0000 7f9429ffb640 1 -- 172.21.15.12:0/1732482044 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f942c0ae9f0 con 0x7f94340ce8e0 2024-04-15T15:23:26.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.335+0000 7f94395ff640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f941806c980 0x7f941806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.335+0000 7f94395ff640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f941806c980 0x7f941806ee40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f94340cc520 tx=0x7f9430007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.339+0000 7f9429ffb640 1 -- 172.21.15.12:0/1732482044 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94340b1b20 con 0x7f94340ce8e0 2024-04-15T15:23:26.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.526+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f943406caf0 con 0x7f94340ce8e0 2024-04-15T15:23:26.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.527+0000 7f9429ffb640 1 -- 172.21.15.12:0/1732482044 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17084 (secure 0 0 0) 0x7f942c07c700 con 0x7f94340ce8e0 2024-04-15T15:23:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:26.530 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":42,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:23:15.001862+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6837/3388147599":"2024-04-16T15:23:15.001839+0000","172.21.15.55:6835/3388147599":"2024-04-16T15:23:15.001839+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f941806c980 msgr2=0x7f941806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f941806c980 0x7f941806ee40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f94340cc520 tx=0x7f9430007e80 comp rx=0 tx=0).stop 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94340ce8e0 msgr2=0x7f94341f7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94340ce8e0 0x7f94341f7520 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f942c009d90 tx=0x7f942c004a40 comp rx=0 tx=0).stop 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 shutdown_connections 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94340ce8e0 0x7f94341f7520 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f941806c980 0x7f941806ee40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943409b0e0 0x7f94341e6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 --2- 172.21.15.12:0/1732482044 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9434071d90 0x7f94341f5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 >> 172.21.15.12:0/1732482044 conn(0x7f94340b3f40 msgr2=0x7f94340af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 shutdown_connections 2024-04-15T15:23:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.529+0000 7f943b863640 1 -- 172.21.15.12:0/1732482044 wait complete. 2024-04-15T15:23:26.543 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:23:26.543 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:23:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.648+0000 7fcfd238f640 1 Processor -- start 2024-04-15T15:23:26.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.649+0000 7fcfd238f640 1 -- start start 2024-04-15T15:23:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.649+0000 7fcfd238f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc15e700 0x7fcfcc0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.649+0000 7fcfd238f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcfcc0d2360 0x7fcfcc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.649+0000 7fcfd238f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfcc15eae0 0x7fcfcc0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.649+0000 7fcfd238f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcfcc0b53b0 con 0x7fcfcc15eae0 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.649+0000 7fcfd238f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fcfcc0b47a0 con 0x7fcfcc0d2360 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.649+0000 7fcfd238f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcfcc0b4a80 con 0x7fcfcc15e700 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfd092c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfcc15eae0 0x7fcfcc0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfd092c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfcc15eae0 0x7fcfcc0d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46100/0 (socket says 172.21.15.12:46100) 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfd092c640 1 -- 172.21.15.12:0/2817593558 learned_addr learned my addr 172.21.15.12:0/2817593558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfc3fff640 1 --2- 172.21.15.12:0/2817593558 >> v2:172.21.15.55:3300/0 conn(0x7fcfcc0d2360 0x7fcfcc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfcbfff640 1 --2- 172.21.15.12:0/2817593558 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc15e700 0x7fcfcc0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfd092c640 1 -- 172.21.15.12:0/2817593558 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc15e700 msgr2=0x7fcfcc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfd092c640 1 --2- 172.21.15.12:0/2817593558 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc15e700 0x7fcfcc0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfd092c640 1 -- 172.21.15.12:0/2817593558 >> v2:172.21.15.55:3300/0 conn(0x7fcfcc0d2360 msgr2=0x7fcfcc0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfd092c640 1 --2- 172.21.15.12:0/2817593558 >> v2:172.21.15.55:3300/0 conn(0x7fcfcc0d2360 0x7fcfcc0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfd092c640 1 -- 172.21.15.12:0/2817593558 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc0b4d60 con 0x7fcfcc15eae0 2024-04-15T15:23:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.650+0000 7fcfc3fff640 1 --2- 172.21.15.12:0/2817593558 >> v2:172.21.15.55:3300/0 conn(0x7fcfcc0d2360 0x7fcfcc0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfd092c640 1 --2- 172.21.15.12:0/2817593558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15eae0 0x7fcfcc0d30a0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0d2100 tx=0x7fcfc402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9ad2f6dcb5c278 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfcb7fe640 1 -- 172.21.15.12:0/2817593558 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc402ca70 con 0x7fcfcc15eae0 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfcb7fe640 1 -- 172.21.15.12:0/2817593558 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc402cc10 con 0x7fcfcc15eae0 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfcb7fe640 1 -- 172.21.15.12:0/2817593558 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40343f0 con 0x7fcfcc15eae0 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfd238f640 1 -- 172.21.15.12:0/2817593558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15eae0 msgr2=0x7fcfcc0d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfd238f640 1 --2- 172.21.15.12:0/2817593558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15eae0 0x7fcfcc0d30a0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0d2100 tx=0x7fcfc402bed0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfd238f640 1 -- 172.21.15.12:0/2817593558 shutdown_connections 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfd238f640 1 --2- 172.21.15.12:0/2817593558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15eae0 0x7fcfcc0d30a0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfd238f640 1 --2- 172.21.15.12:0/2817593558 >> v2:172.21.15.55:3300/0 conn(0x7fcfcc0d2360 0x7fcfcc0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfd238f640 1 --2- 172.21.15.12:0/2817593558 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc15e700 0x7fcfcc0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.651+0000 7fcfd238f640 1 -- 172.21.15.12:0/2817593558 >> 172.21.15.12:0/2817593558 conn(0x7fcfcc08bf60 msgr2=0x7fcfcc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:26.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.652+0000 7fcfd238f640 1 -- 172.21.15.12:0/2817593558 shutdown_connections 2024-04-15T15:23:26.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.652+0000 7fcfd238f640 1 -- 172.21.15.12:0/2817593558 wait complete. 2024-04-15T15:23:26.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.653+0000 7fcfd238f640 1 Processor -- start 2024-04-15T15:23:26.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.653+0000 7fcfd238f640 1 -- start start 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.653+0000 7fcfd238f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc0d2360 0x7fcfcc154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.653+0000 7fcfd238f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15e700 0x7fcfcc161570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.653+0000 7fcfd238f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfcc15eae0 0x7fcfcc0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.653+0000 7fcfd238f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcfcc0b5d80 con 0x7fcfcc15e700 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.653+0000 7fcfd238f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcfcc0b4450 con 0x7fcfcc15eae0 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.653+0000 7fcfd238f640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcfcc0d1d40 con 0x7fcfcc0d2360 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15e700 0x7fcfcc161570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15e700 0x7fcfcc161570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46116/0 (socket says 172.21.15.12:46116) 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 -- 172.21.15.12:0/1501518966 learned_addr learned my addr 172.21.15.12:0/1501518966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfd092c640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfcc15eae0 0x7fcfcc0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfcbfff640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc0d2360 0x7fcfcc154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 -- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc0d2360 msgr2=0x7fcfcc154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc0d2360 0x7fcfcc154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 -- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfcc15eae0 msgr2=0x7fcfcc0c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfcc15eae0 0x7fcfcc0c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 -- 172.21.15.12:0/1501518966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc160d90 con 0x7fcfcc15e700 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfd092c640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfcc15eae0 0x7fcfcc0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc3fff640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15e700 0x7fcfcc161570 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400d800 tx=0x7fcfb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc97fa640 1 -- 172.21.15.12:0/1501518966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4018070 con 0x7fcfcc15e700 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc97fa640 1 -- 172.21.15.12:0/1501518966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfb4004030 con 0x7fcfcc15e700 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfcc0cc460 con 0x7fcfcc15e700 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfc97fa640 1 -- 172.21.15.12:0/1501518966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4004400 con 0x7fcfcc15e700 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.654+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfcc161050 con 0x7fcfcc15e700 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.655+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfcc0689e0 con 0x7fcfcc15e700 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.656+0000 7fcfc97fa640 1 -- 172.21.15.12:0/1501518966 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fcfb40096a0 con 0x7fcfcc15e700 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.656+0000 7fcfc97fa640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcfac06c980 0x7fcfac06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.656+0000 7fcfc97fa640 1 -- 172.21.15.12:0/1501518966 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fcfb408c0c0 con 0x7fcfcc15e700 2024-04-15T15:23:26.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.657+0000 7fcfcbfff640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcfac06c980 0x7fcfac06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.657+0000 7fcfcbfff640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcfac06c980 0x7fcfac06ee40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc155490 tx=0x7fcfbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.661+0000 7fcfc97fa640 1 -- 172.21.15.12:0/1501518966 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfb405a300 con 0x7fcfcc15e700 2024-04-15T15:23:26.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.844+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcfcc0d0960 con 0x7fcfcc15e700 2024-04-15T15:23:26.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.844+0000 7fcfc97fa640 1 -- 172.21.15.12:0/1501518966 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fcfb4059c00 con 0x7fcfcc15e700 2024-04-15T15:23:26.847 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcfac06c980 msgr2=0x7fcfac06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcfac06c980 0x7fcfac06ee40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc155490 tx=0x7fcfbc008040 comp rx=0 tx=0).stop 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15e700 msgr2=0x7fcfcc161570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15e700 0x7fcfcc161570 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400d800 tx=0x7fcfb400dcc0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 shutdown_connections 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fcfac06c980 0x7fcfac06ee40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfcc15eae0 0x7fcfcc0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfcc15e700 0x7fcfcc161570 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 --2- 172.21.15.12:0/1501518966 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcfcc0d2360 0x7fcfcc154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 >> 172.21.15.12:0/1501518966 conn(0x7fcfcc08bf60 msgr2=0x7fcfcc0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:26.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 shutdown_connections 2024-04-15T15:23:26.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.846+0000 7fcfd238f640 1 -- 172.21.15.12:0/1501518966 wait complete. 2024-04-15T15:23:26.860 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:23:26.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.965+0000 7f88f180d640 1 Processor -- start 2024-04-15T15:23:26.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.965+0000 7f88f180d640 1 -- start start 2024-04-15T15:23:26.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88f180d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f88ec15e700 0x7f88ec0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88f180d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec0d2360 0x7f88ec0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88f180d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88ec0b47a0 con 0x7f88ec15eae0 2024-04-15T15:23:26.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88f180d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f88ec0b53b0 con 0x7f88ec15e700 2024-04-15T15:23:26.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88f180d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f88ec0b4a80 con 0x7f88ec0d2360 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88eb7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f88ec15eae0 0x7f88ec0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47070/0 (socket says 172.21.15.12:47070) 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88eb7fe640 1 -- 172.21.15.12:0/2002663908 learned_addr learned my addr 172.21.15.12:0/2002663908 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88eaffd640 1 --2- 172.21.15.12:0/2002663908 >> v2:172.21.15.55:3300/0 conn(0x7f88ec15e700 0x7f88ec0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.966+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1437691922 0 0) 0x7f88ec0b47a0 con 0x7f88ec15eae0 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d8003520 con 0x7f88ec15eae0 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88ea7fc640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec0d2360 0x7f88ec0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 916563031 0 0) 0x7f88d8003520 con 0x7f88ec15eae0 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec0d2360 msgr2=0x7f88ec0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec0d2360 0x7f88ec0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 >> v2:172.21.15.55:3300/0 conn(0x7f88ec15e700 msgr2=0x7f88ec0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 --2- 172.21.15.12:0/2002663908 >> v2:172.21.15.55:3300/0 conn(0x7f88ec15e700 0x7f88ec0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec0b4d60 con 0x7f88ec15eae0 2024-04-15T15:23:26.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f88dc0019f0 con 0x7f88ec15eae0 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 >> v1:172.21.15.12:6789/0 conn(0x7f88ec15eae0 legacy=0x7f88ec0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec0d2360 0x7f88d80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8003f30 0x7f88d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88d8004d40 0x7f88d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec0b53b0 con 0x7f88d8004d40 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec0b4a80 con 0x7f88d8003f30 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f88d800b9c0 con 0x7f88ec0d2360 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88eb7fe640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88d8004d40 0x7f88d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88eaffd640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec0d2360 0x7f88d80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88ea7fc640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8003f30 0x7f88d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88eb7fe640 1 -- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec0d2360 msgr2=0x7f88d80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88eb7fe640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec0d2360 0x7f88d80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88eb7fe640 1 -- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8003f30 msgr2=0x7f88d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:26.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88eb7fe640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8003f30 0x7f88d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88eb7fe640 1 -- 172.21.15.12:0/2002663908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d8003520 con 0x7f88d8004d40 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.967+0000 7f88ea7fc640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8003f30 0x7f88d8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.968+0000 7f88eb7fe640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88d8004d40 0x7f88d80051c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00a880 tx=0x7f88d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d09955a2c359ce server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.968+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc00a4a0 con 0x7f88d8004d40 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.968+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88dc034650 con 0x7f88d8004d40 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.968+0000 7f88e9ffb640 1 -- 172.21.15.12:0/2002663908 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc034a40 con 0x7f88d8004d40 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.969+0000 7f88f180d640 1 -- 172.21.15.12:0/2002663908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88d8004d40 msgr2=0x7f88d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.969+0000 7f88f180d640 1 --2- 172.21.15.12:0/2002663908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88d8004d40 0x7f88d80051c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00a880 tx=0x7f88d8005830 comp rx=0 tx=0).stop 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.969+0000 7f88eb7fe640 1 -- 172.21.15.12:0/2002663908 reap_dead start 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.969+0000 7f88f180d640 1 -- 172.21.15.12:0/2002663908 shutdown_connections 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.969+0000 7f88f180d640 1 -- 172.21.15.12:0/2002663908 >> 172.21.15.12:0/2002663908 conn(0x7f88ec08bf60 msgr2=0x7f88ec0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.969+0000 7f88f180d640 1 -- 172.21.15.12:0/2002663908 shutdown_connections 2024-04-15T15:23:26.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.969+0000 7f88f180d640 1 -- 172.21.15.12:0/2002663908 wait complete. 2024-04-15T15:23:26.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.970+0000 7f88f180d640 1 Processor -- start 2024-04-15T15:23:26.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.970+0000 7f88f180d640 1 -- start start 2024-04-15T15:23:26.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88f180d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8004d40 0x7f88ec160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88f180d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88ec0c9e30 0x7f88ec1f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88f180d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec06daa0 0x7f88ec06df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88f180d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec0b5d80 con 0x7f88ec0c9e30 2024-04-15T15:23:26.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88f180d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec0b4450 con 0x7f88d8004d40 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88f180d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f88ec0d1d40 con 0x7f88ec06daa0 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88ec0c9e30 0x7f88ec1f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88ec0c9e30 0x7f88ec1f74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46132/0 (socket says 172.21.15.12:46132) 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 -- 172.21.15.12:0/1877394361 learned_addr learned my addr 172.21.15.12:0/1877394361 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88eaffd640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8004d40 0x7f88ec160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88eb7fe640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec06daa0 0x7f88ec06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 -- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec06daa0 msgr2=0x7f88ec06df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec06daa0 0x7f88ec06df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 -- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8004d40 msgr2=0x7f88ec160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8004d40 0x7f88ec160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 -- 172.21.15.12:0/1877394361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec0b7740 con 0x7f88ec0c9e30 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.971+0000 7f88ea7fc640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88ec0c9e30 0x7f88ec1f74a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f88d4011570 tx=0x7f88d4011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.972+0000 7f88e3fff640 1 -- 172.21.15.12:0/1877394361 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4015870 con 0x7f88ec0c9e30 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.972+0000 7f88e3fff640 1 -- 172.21.15.12:0/1877394361 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d4015a10 con 0x7f88ec0c9e30 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.972+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ec160560 con 0x7f88ec0c9e30 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.972+0000 7f88e3fff640 1 -- 172.21.15.12:0/1877394361 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4015dc0 con 0x7f88ec0c9e30 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.972+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ec160ae0 con 0x7f88ec0c9e30 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.973+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88ec0689e0 con 0x7f88ec0c9e30 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.973+0000 7f88e3fff640 1 -- 172.21.15.12:0/1877394361 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f88d401d400 con 0x7f88ec0c9e30 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.973+0000 7f88e3fff640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f88cc06c980 0x7f88cc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.974+0000 7f88e3fff640 1 -- 172.21.15.12:0/1877394361 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f88d408c620 con 0x7f88ec0c9e30 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.974+0000 7f88eaffd640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f88cc06c980 0x7f88cc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.974+0000 7f88eaffd640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f88cc06c980 0x7f88cc06ee40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f88ec06e690 tx=0x7f88e4009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:26.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:26.979+0000 7f88e3fff640 1 -- 172.21.15.12:0/1877394361 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f88d4019030 con 0x7f88ec0c9e30 2024-04-15T15:23:27.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.164+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f88ec0ba500 con 0x7f88ec0c9e30 2024-04-15T15:23:27.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.164+0000 7f88e3fff640 1 -- 172.21.15.12:0/1877394361 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f88d405a770 con 0x7f88ec0c9e30 2024-04-15T15:23:27.167 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:23:27.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f88cc06c980 msgr2=0x7f88cc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f88cc06c980 0x7f88cc06ee40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f88ec06e690 tx=0x7f88e4009110 comp rx=0 tx=0).stop 2024-04-15T15:23:27.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88ec0c9e30 msgr2=0x7f88ec1f74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88ec0c9e30 0x7f88ec1f74a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f88d4011570 tx=0x7f88d4011970 comp rx=0 tx=0).stop 2024-04-15T15:23:27.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 shutdown_connections 2024-04-15T15:23:27.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f88ec06daa0 0x7f88ec06df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88ec0c9e30 0x7f88ec1f74a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f88cc06c980 0x7f88cc06ee40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 --2- 172.21.15.12:0/1877394361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d8004d40 0x7f88ec160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.166+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 >> 172.21.15.12:0/1877394361 conn(0x7f88ec08bf60 msgr2=0x7f88ec0d0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:27.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.167+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 shutdown_connections 2024-04-15T15:23:27.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.167+0000 7f88f180d640 1 -- 172.21.15.12:0/1877394361 wait complete. 2024-04-15T15:23:27.180 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:23:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.285+0000 7f58967a8640 1 Processor -- start 2024-04-15T15:23:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.285+0000 7f58967a8640 1 -- start start 2024-04-15T15:23:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f58967a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890070940 0x7f589009b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f58967a8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5890160310 0x7f5890070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f58967a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58900d05e0 con 0x7f58900d0da0 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f58967a8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5890071260 con 0x7f5890160310 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f58967a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f58900d1940 con 0x7f5890070940 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f5887fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5890160310 0x7f5890070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f588ffff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890070940 0x7f589009b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f5887fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5890160310 0x7f5890070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:46170/0 (socket says 172.21.15.12:46170) 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f5894d45640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f58900d0da0 0x7f58900d1230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:42746/0 (socket says 172.21.15.12:42746) 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.286+0000 7f5887fff640 1 -- 172.21.15.12:0/709958060 learned_addr learned my addr 172.21.15.12:0/709958060 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.287+0000 7f5887fff640 1 -- 172.21.15.12:0/709958060 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890070940 msgr2=0x7f589009b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.287+0000 7f5887fff640 1 --2- 172.21.15.12:0/709958060 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890070940 0x7f589009b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.287+0000 7f5887fff640 1 -- 172.21.15.12:0/709958060 >> v1:172.21.15.55:6789/0 conn(0x7f58900d0da0 legacy=0x7f58900d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.287+0000 7f5887fff640 1 -- 172.21.15.12:0/709958060 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589015f720 con 0x7f5890160310 2024-04-15T15:23:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.287+0000 7f588ffff640 1 --2- 172.21.15.12:0/709958060 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890070940 0x7f589009b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f5887fff640 1 --2- 172.21.15.12:0/709958060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890160310 0x7f5890070d20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5878009870 tx=0x7f587802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db3ed0f30518309f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f588f7fe640 1 -- 172.21.15.12:0/709958060 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878034070 con 0x7f5890160310 2024-04-15T15:23:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f588f7fe640 1 -- 172.21.15.12:0/709958060 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878030840 con 0x7f5890160310 2024-04-15T15:23:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f588f7fe640 1 -- 172.21.15.12:0/709958060 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878030c10 con 0x7f5890160310 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f58967a8640 1 -- 172.21.15.12:0/709958060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890160310 msgr2=0x7f5890070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f58967a8640 1 --2- 172.21.15.12:0/709958060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890160310 0x7f5890070d20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5878009870 tx=0x7f587802b980 comp rx=0 tx=0).stop 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f58967a8640 1 -- 172.21.15.12:0/709958060 shutdown_connections 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f58967a8640 1 --2- 172.21.15.12:0/709958060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890160310 0x7f5890070d20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f58967a8640 1 --2- 172.21.15.12:0/709958060 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890070940 0x7f589009b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.288+0000 7f58967a8640 1 -- 172.21.15.12:0/709958060 >> 172.21.15.12:0/709958060 conn(0x7f58900b3f40 msgr2=0x7f58900b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.289+0000 7f58967a8640 1 -- 172.21.15.12:0/709958060 shutdown_connections 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.289+0000 7f58967a8640 1 -- 172.21.15.12:0/709958060 wait complete. 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.290+0000 7f58967a8640 1 Processor -- start 2024-04-15T15:23:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.290+0000 7f58967a8640 1 -- start start 2024-04-15T15:23:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.290+0000 7f58967a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890070940 0x7f5890148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.290+0000 7f58967a8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58900d0da0 0x7f589014d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.290+0000 7f58967a8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890160310 0x7f589014dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.290+0000 7f58967a8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f58900d0880 con 0x7f58900d0da0 2024-04-15T15:23:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.290+0000 7f58967a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f589015fce0 con 0x7f5890070940 2024-04-15T15:23:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.290+0000 7f58967a8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f5890160770 con 0x7f5890160310 2024-04-15T15:23:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58900d0da0 0x7f589014d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58900d0da0 0x7f589014d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46142/0 (socket says 172.21.15.12:46142) 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 -- 172.21.15.12:0/1821906967 learned_addr learned my addr 172.21.15.12:0/1821906967 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f588ffff640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890070940 0x7f5890148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5894d45640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890160310 0x7f589014dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 -- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890160310 msgr2=0x7f589014dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890160310 0x7f589014dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 -- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890070940 msgr2=0x7f5890148320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890070940 0x7f5890148320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 -- 172.21.15.12:0/1821906967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5890148860 con 0x7f58900d0da0 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f5887fff640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58900d0da0 0x7f589014d830 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5878000c00 tx=0x7f5878004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f588d7fa640 1 -- 172.21.15.12:0/1821906967 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878048070 con 0x7f58900d0da0 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f588d7fa640 1 -- 172.21.15.12:0/1821906967 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878030640 con 0x7f58900d0da0 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f589015de00 con 0x7f58900d0da0 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f588d7fa640 1 -- 172.21.15.12:0/1821906967 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878034040 con 0x7f58900d0da0 2024-04-15T15:23:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.291+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58901595c0 con 0x7f58900d0da0 2024-04-15T15:23:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.292+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58900689e0 con 0x7f58900d0da0 2024-04-15T15:23:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.293+0000 7f588d7fa640 1 -- 172.21.15.12:0/1821906967 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f587804b070 con 0x7f58900d0da0 2024-04-15T15:23:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.293+0000 7f588d7fa640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f587006c9b0 0x7f587006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.293+0000 7f588d7fa640 1 -- 172.21.15.12:0/1821906967 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f58780adc80 con 0x7f58900d0da0 2024-04-15T15:23:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.294+0000 7f588ffff640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f587006c9b0 0x7f587006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.294+0000 7f588ffff640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f587006c9b0 0x7f587006ee70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5880009560 tx=0x7f5880009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.299+0000 7f588d7fa640 1 -- 172.21.15.12:0/1821906967 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f587807b7f0 con 0x7f58900d0da0 2024-04-15T15:23:27.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.481+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5890071910 con 0x7f58900d0da0 2024-04-15T15:23:27.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.481+0000 7f588d7fa640 1 -- 172.21.15.12:0/1821906967 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f587807faf0 con 0x7f58900d0da0 2024-04-15T15:23:27.484 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:23:27.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.483+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f587006c9b0 msgr2=0x7f587006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.483+0000 7f58967a8640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f587006c9b0 0x7f587006ee70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5880009560 tx=0x7f5880009190 comp rx=0 tx=0).stop 2024-04-15T15:23:27.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.483+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58900d0da0 msgr2=0x7f589014d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.483+0000 7f58967a8640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58900d0da0 0x7f589014d830 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5878000c00 tx=0x7f5878004730 comp rx=0 tx=0).stop 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.484+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 shutdown_connections 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.484+0000 7f58967a8640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f5890160310 0x7f589014dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.484+0000 7f58967a8640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58900d0da0 0x7f589014d830 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.484+0000 7f58967a8640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f587006c9b0 0x7f587006ee70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.484+0000 7f58967a8640 1 --2- 172.21.15.12:0/1821906967 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5890070940 0x7f5890148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.484+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 >> 172.21.15.12:0/1821906967 conn(0x7f58900b3f40 msgr2=0x7f589015ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.484+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 shutdown_connections 2024-04-15T15:23:27.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.484+0000 7f58967a8640 1 -- 172.21.15.12:0/1821906967 wait complete. 2024-04-15T15:23:27.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:23:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.600+0000 7f10587f4640 1 Processor -- start 2024-04-15T15:23:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.600+0000 7f10587f4640 1 -- start start 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.601+0000 7f10587f4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1050070940 0x7f105009b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.601+0000 7f10587f4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10500d05e0 con 0x7f10500d0da0 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.601+0000 7f10587f4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105015f720 con 0x7f1050160310 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.601+0000 7f10587f4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f105015fa00 con 0x7f1050070940 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.601+0000 7f1056590640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1050070940 0x7f105009b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.601+0000 7f1056590640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1050070940 0x7f105009b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46154/0 (socket says 172.21.15.12:46154) 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.601+0000 7f1056590640 1 -- 172.21.15.12:0/1710264317 learned_addr learned my addr 172.21.15.12:0/1710264317 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.602+0000 7f1056590640 1 -- 172.21.15.12:0/1710264317 >> v1:172.21.15.55:6789/0 conn(0x7f1050160310 legacy=0x7f1050070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.602+0000 7f1056590640 1 -- 172.21.15.12:0/1710264317 >> v1:172.21.15.12:6789/0 conn(0x7f10500d0da0 legacy=0x7f10500d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.602+0000 7f1056590640 1 -- 172.21.15.12:0/1710264317 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105015fce0 con 0x7f1050070940 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.602+0000 7f1056590640 1 --2- 172.21.15.12:0/1710264317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050070940 0x7f105009b880 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1048009870 tx=0x7f104802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc408a21f15a221e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f105558e640 1 -- 172.21.15.12:0/1710264317 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048034070 con 0x7f1050070940 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f105558e640 1 -- 172.21.15.12:0/1710264317 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1048030860 con 0x7f1050070940 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f105558e640 1 -- 172.21.15.12:0/1710264317 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048030c30 con 0x7f1050070940 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f10587f4640 1 -- 172.21.15.12:0/1710264317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050070940 msgr2=0x7f105009b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f10587f4640 1 --2- 172.21.15.12:0/1710264317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050070940 0x7f105009b880 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1048009870 tx=0x7f104802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f10587f4640 1 -- 172.21.15.12:0/1710264317 shutdown_connections 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f10587f4640 1 --2- 172.21.15.12:0/1710264317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050070940 0x7f105009b880 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f10587f4640 1 -- 172.21.15.12:0/1710264317 >> 172.21.15.12:0/1710264317 conn(0x7f10500b3f40 msgr2=0x7f10500b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f10587f4640 1 -- 172.21.15.12:0/1710264317 shutdown_connections 2024-04-15T15:23:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.603+0000 7f10587f4640 1 -- 172.21.15.12:0/1710264317 wait complete. 2024-04-15T15:23:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.604+0000 7f10587f4640 1 Processor -- start 2024-04-15T15:23:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.604+0000 7f10587f4640 1 -- start start 2024-04-15T15:23:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f10587f4640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1050070940 0x7f10500b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f10587f4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10500d0da0 0x7f10500b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f10587f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050160310 0x7f10500b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f10587f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1050160770 con 0x7f1050160310 2024-04-15T15:23:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f10587f4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f10500d1900 con 0x7f10500d0da0 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f10587f4640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f105015f0d0 con 0x7f1050070940 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1056d91640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050160310 0x7f10500b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1056d91640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050160310 0x7f10500b57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46160/0 (socket says 172.21.15.12:46160) 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1056d91640 1 -- 172.21.15.12:0/3665709354 learned_addr learned my addr 172.21.15.12:0/3665709354 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1055d8f640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10500d0da0 0x7f10500b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1056590640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1050070940 0x7f10500b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1056d91640 1 -- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1050070940 msgr2=0x7f10500b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1056d91640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1050070940 0x7f10500b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1056d91640 1 -- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10500d0da0 msgr2=0x7f10500b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.605+0000 7f1056d91640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10500d0da0 0x7f10500b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.606+0000 7f1056d91640 1 -- 172.21.15.12:0/3665709354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10500b5ef0 con 0x7f1050160310 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.606+0000 7f1056d91640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050160310 0x7f10500b57e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f104000c710 tx=0x7f104000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.606+0000 7f104effd640 1 -- 172.21.15.12:0/3665709354 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1040016020 con 0x7f1050160310 2024-04-15T15:23:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.606+0000 7f104effd640 1 -- 172.21.15.12:0/3665709354 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1040006d50 con 0x7f1050160310 2024-04-15T15:23:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.606+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10500b61d0 con 0x7f1050160310 2024-04-15T15:23:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.606+0000 7f104effd640 1 -- 172.21.15.12:0/3665709354 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1040003bb0 con 0x7f1050160310 2024-04-15T15:23:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.606+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10500b6790 con 0x7f1050160310 2024-04-15T15:23:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.607+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10500689e0 con 0x7f1050160310 2024-04-15T15:23:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.607+0000 7f104effd640 1 -- 172.21.15.12:0/3665709354 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f104000a850 con 0x7f1050160310 2024-04-15T15:23:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.608+0000 7f104effd640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f103c06c980 0x7f103c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.608+0000 7f104effd640 1 -- 172.21.15.12:0/3665709354 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f104008b9f0 con 0x7f1050160310 2024-04-15T15:23:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.608+0000 7f1056590640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f103c06c980 0x7f103c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.608+0000 7f1056590640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f103c06c980 0x7f103c06ee40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1048036000 tx=0x7f104803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.613+0000 7f104effd640 1 -- 172.21.15.12:0/3665709354 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1040020020 con 0x7f1050160310 2024-04-15T15:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.808+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1050070720 con 0x7f1050160310 2024-04-15T15:23:27.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.808+0000 7f104effd640 1 -- 172.21.15.12:0/3665709354 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7f1040019310 con 0x7f1050160310 2024-04-15T15:23:27.811 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.55:6837/3388147599 2024-04-16T15:23:15.001839+0000 2024-04-15T15:23:27.811 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.55:6835/3388147599 2024-04-16T15:23:15.001839+0000 2024-04-15T15:23:27.811 INFO:teuthology.orchestra.run.smithi012.stderr:listed 2 entries 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f103c06c980 msgr2=0x7f103c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f103c06c980 0x7f103c06ee40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1048036000 tx=0x7f104803b040 comp rx=0 tx=0).stop 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050160310 msgr2=0x7f10500b57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050160310 0x7f10500b57e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f104000c710 tx=0x7f104000cbd0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 shutdown_connections 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1050160310 0x7f10500b57e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f103c06c980 0x7f103c06ee40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10500d0da0 0x7f10500b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 --2- 172.21.15.12:0/3665709354 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f1050070940 0x7f10500b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 >> 172.21.15.12:0/3665709354 conn(0x7f10500b3f40 msgr2=0x7f10500b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 shutdown_connections 2024-04-15T15:23:27.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:27.811+0000 7f10587f4640 1 -- 172.21.15.12:0/3665709354 wait complete. 2024-04-15T15:23:27.823 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi055.front.sepia.ceph.com... 2024-04-15T15:23:27.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:27.824 DEBUG:teuthology.orchestra.run.smithi055:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-15T15:23:27.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi055.stderr:ceph-fuse[46473]: fuse finished with error 0 and tester_r 0 2024-04-15T15:23:27.858 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:23:27.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:27.859 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:23:28.030 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:23:28.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.133+0000 7f0f48315640 1 Processor -- start 2024-04-15T15:23:28.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.133+0000 7f0f48315640 1 -- start start 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f48315640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f4015f0d0 0x7f0f400d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f48315640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f400d05e0 con 0x7f0f4009b880 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f48315640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f4015f720 con 0x7f0f40070940 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f48315640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4009bce0 con 0x7f0f4015f0d0 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f458b0640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0f4009b880 0x7f0f40070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47090/0 (socket says 172.21.15.12:47090) 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f458b0640 1 -- 172.21.15.12:0/4065871721 learned_addr learned my addr 172.21.15.12:0/4065871721 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f468b2640 1 --2- 172.21.15.12:0/4065871721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f4015f0d0 0x7f0f400d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f450af640 1 -- 172.21.15.12:0/4065871721 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3967977837 0 0) 0x7f0f400d05e0 con 0x7f0f4009b880 2024-04-15T15:23:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f450af640 1 -- 172.21.15.12:0/4065871721 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f2c003520 con 0x7f0f4009b880 2024-04-15T15:23:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f450af640 1 -- 172.21.15.12:0/4065871721 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3278962855 0 0) 0x7f0f4015f720 con 0x7f0f40070940 2024-04-15T15:23:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f450af640 1 -- 172.21.15.12:0/4065871721 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f400d05e0 con 0x7f0f40070940 2024-04-15T15:23:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f468b2640 1 -- 172.21.15.12:0/4065871721 >> v1:172.21.15.55:6789/0 conn(0x7f0f40070940 legacy=0x7f0f40160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f468b2640 1 -- 172.21.15.12:0/4065871721 >> v1:172.21.15.12:6789/0 conn(0x7f0f4009b880 legacy=0x7f0f40070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f450af640 1 -- 172.21.15.12:0/4065871721 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3553779446 0 0) 0x7f0f2c003520 con 0x7f0f4009b880 2024-04-15T15:23:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.134+0000 7f0f468b2640 1 -- 172.21.15.12:0/4065871721 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4015fa00 con 0x7f0f4015f0d0 2024-04-15T15:23:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.135+0000 7f0f468b2640 1 --2- 172.21.15.12:0/4065871721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f4015f0d0 0x7f0f400d0da0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009dc0 tx=0x7f0f3802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f4bc993540f20e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.135+0000 7f0f450af640 1 -- 172.21.15.12:0/4065871721 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f38039070 con 0x7f0f4015f0d0 2024-04-15T15:23:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f450af640 1 -- 172.21.15.12:0/4065871721 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f3802c910 con 0x7f0f4015f0d0 2024-04-15T15:23:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f450af640 1 -- 172.21.15.12:0/4065871721 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f30001670 con 0x7f0f4015f0d0 2024-04-15T15:23:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f48315640 1 -- 172.21.15.12:0/4065871721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f4015f0d0 msgr2=0x7f0f400d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f48315640 1 --2- 172.21.15.12:0/4065871721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f4015f0d0 0x7f0f400d0da0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009dc0 tx=0x7f0f3802bde0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f48315640 1 -- 172.21.15.12:0/4065871721 shutdown_connections 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f48315640 1 --2- 172.21.15.12:0/4065871721 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f4015f0d0 0x7f0f400d0da0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f48315640 1 -- 172.21.15.12:0/4065871721 >> 172.21.15.12:0/4065871721 conn(0x7f0f400b3f40 msgr2=0x7f0f400b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f48315640 1 -- 172.21.15.12:0/4065871721 shutdown_connections 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.136+0000 7f0f48315640 1 -- 172.21.15.12:0/4065871721 wait complete. 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.137+0000 7f0f48315640 1 Processor -- start 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.137+0000 7f0f48315640 1 -- start start 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f48315640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f40070940 0x7f0f4015b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f48315640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f4009b880 0x7f0f401564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f48315640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f4015f0d0 0x7f0f400b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f48315640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0f400d12e0 con 0x7f0f4015f0d0 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f48315640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0f400d0880 con 0x7f0f4009b880 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f48315640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0f400d1460 con 0x7f0f40070940 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f468b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f4015f0d0 0x7f0f400b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f468b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f4015f0d0 0x7f0f400b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46176/0 (socket says 172.21.15.12:46176) 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f468b2640 1 -- 172.21.15.12:0/941071792 learned_addr learned my addr 172.21.15.12:0/941071792 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f458b0640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f4009b880 0x7f0f401564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f460b1640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f40070940 0x7f0f4015b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f468b2640 1 -- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f40070940 msgr2=0x7f0f4015b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f468b2640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f40070940 0x7f0f4015b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f468b2640 1 -- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f4009b880 msgr2=0x7f0f401564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f468b2640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f4009b880 0x7f0f401564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.138+0000 7f0f468b2640 1 -- 172.21.15.12:0/941071792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f400b6230 con 0x7f0f4015f0d0 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.139+0000 7f0f468b2640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f4015f0d0 0x7f0f400b5a00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009ee0 tx=0x7f0f38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.139+0000 7f0f36ffd640 1 -- 172.21.15.12:0/941071792 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f38039050 con 0x7f0f4015f0d0 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.139+0000 7f0f36ffd640 1 -- 172.21.15.12:0/941071792 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f38045070 con 0x7f0f4015f0d0 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.139+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f40156a20 con 0x7f0f4015f0d0 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.139+0000 7f0f36ffd640 1 -- 172.21.15.12:0/941071792 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f38009450 con 0x7f0f4015f0d0 2024-04-15T15:23:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.139+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f4015bb50 con 0x7f0f4015f0d0 2024-04-15T15:23:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.140+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f400cea70 con 0x7f0f4015f0d0 2024-04-15T15:23:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.140+0000 7f0f36ffd640 1 -- 172.21.15.12:0/941071792 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0f38004690 con 0x7f0f4015f0d0 2024-04-15T15:23:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.141+0000 7f0f36ffd640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f2406c980 0x7f0f2406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.141+0000 7f0f36ffd640 1 -- 172.21.15.12:0/941071792 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0f380ae5d0 con 0x7f0f4015f0d0 2024-04-15T15:23:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.141+0000 7f0f460b1640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f2406c980 0x7f0f2406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.143+0000 7f0f460b1640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f2406c980 0x7f0f2406ee40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0011e0 tx=0x7f0f3c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.146+0000 7f0f36ffd640 1 -- 172.21.15.12:0/941071792 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f400cea70 con 0x7f0f4015f0d0 2024-04-15T15:23:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.371+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f400689e0 con 0x7f0f4015f0d0 2024-04-15T15:23:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.372+0000 7f0f36ffd640 1 -- 172.21.15.12:0/941071792 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5180 (secure 0 0 0) 0x7f0f3807c8a0 con 0x7f0f4015f0d0 2024-04-15T15:23:28.374 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:28.375 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5102,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/852515692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":852515692},{"type":"v1","addr":"172.21.15.55:6837","nonce":852515692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:23:19.042212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5200":{"gid":5200,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5200,"qdb_cluster":[5200]},"id":2}]} 2024-04-15T15:23:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-15T15:23:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f2406c980 msgr2=0x7f0f2406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f2406c980 0x7f0f2406ee40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0011e0 tx=0x7f0f3c009040 comp rx=0 tx=0).stop 2024-04-15T15:23:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f4015f0d0 msgr2=0x7f0f400b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f4015f0d0 0x7f0f400b5a00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009ee0 tx=0x7f0f38004a40 comp rx=0 tx=0).stop 2024-04-15T15:23:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 shutdown_connections 2024-04-15T15:23:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f4015f0d0 0x7f0f400b5a00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0f2406c980 0x7f0f2406ee40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f4009b880 0x7f0f401564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 --2- 172.21.15.12:0/941071792 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0f40070940 0x7f0f4015b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 >> 172.21.15.12:0/941071792 conn(0x7f0f400b3f40 msgr2=0x7f0f40074ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 shutdown_connections 2024-04-15T15:23:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.374+0000 7f0f48315640 1 -- 172.21.15.12:0/941071792 wait complete. 2024-04-15T15:23:28.387 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:23:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0914ca5640 1 Processor -- start 2024-04-15T15:23:28.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0914ca5640 1 -- start start 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0914ca5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0910158770 0x7f091015bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0914ca5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0910072e30 0x7f091006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0914ca5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f091015c1d0 con 0x7f09100b5230 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0914ca5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f09100b4da0 con 0x7f0910072e30 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0914ca5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f09100b4190 con 0x7f0910158770 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f090ed9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0910072e30 0x7f091006dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f090e59c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0910158770 0x7f091015bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0907fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f09100b5230 0x7f0910072720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:42776/0 (socket says 172.21.15.12:42776) 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.490+0000 7f0907fff640 1 -- 172.21.15.12:0/1561538690 learned_addr learned my addr 172.21.15.12:0/1561538690 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.491+0000 7f090e59c640 1 -- 172.21.15.12:0/1561538690 >> v2:172.21.15.55:3300/0 conn(0x7f0910072e30 msgr2=0x7f091006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:28.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.491+0000 7f090e59c640 1 --2- 172.21.15.12:0/1561538690 >> v2:172.21.15.55:3300/0 conn(0x7f0910072e30 0x7f091006dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.491+0000 7f090e59c640 1 -- 172.21.15.12:0/1561538690 >> v1:172.21.15.55:6789/0 conn(0x7f09100b5230 legacy=0x7f0910072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.491+0000 7f090e59c640 1 -- 172.21.15.12:0/1561538690 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09100b4470 con 0x7f0910158770 2024-04-15T15:23:28.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.491+0000 7f090dd9b640 1 -- 172.21.15.12:0/1561538690 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150862373 0 0) 0x7f08f8001110 con 0x7f09100b5230 2024-04-15T15:23:28.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.491+0000 7f090ed9d640 1 --2- 172.21.15.12:0/1561538690 >> v2:172.21.15.55:3300/0 conn(0x7f0910072e30 0x7f091006dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:28.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.491+0000 7f090e59c640 1 --2- 172.21.15.12:0/1561538690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0910158770 0x7f091015bc90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09000097a0 tx=0x7f090002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29dc58bc5b1aca3a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f090dd9b640 1 -- 172.21.15.12:0/1561538690 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900035020 con 0x7f0910158770 2024-04-15T15:23:28.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f090dd9b640 1 -- 172.21.15.12:0/1561538690 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0900031830 con 0x7f0910158770 2024-04-15T15:23:28.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f090dd9b640 1 -- 172.21.15.12:0/1561538690 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900031c00 con 0x7f0910158770 2024-04-15T15:23:28.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f0914ca5640 1 -- 172.21.15.12:0/1561538690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0910158770 msgr2=0x7f091015bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f0914ca5640 1 --2- 172.21.15.12:0/1561538690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0910158770 0x7f091015bc90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09000097a0 tx=0x7f090002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f0914ca5640 1 -- 172.21.15.12:0/1561538690 shutdown_connections 2024-04-15T15:23:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f0914ca5640 1 --2- 172.21.15.12:0/1561538690 >> v2:172.21.15.55:3300/0 conn(0x7f0910072e30 0x7f091006dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f0914ca5640 1 --2- 172.21.15.12:0/1561538690 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0910158770 0x7f091015bc90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f0914ca5640 1 -- 172.21.15.12:0/1561538690 >> 172.21.15.12:0/1561538690 conn(0x7f091006be00 msgr2=0x7f091006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f0914ca5640 1 -- 172.21.15.12:0/1561538690 shutdown_connections 2024-04-15T15:23:28.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.492+0000 7f0914ca5640 1 -- 172.21.15.12:0/1561538690 wait complete. 2024-04-15T15:23:28.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.493+0000 7f0914ca5640 1 Processor -- start 2024-04-15T15:23:28.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.493+0000 7f0914ca5640 1 -- start start 2024-04-15T15:23:28.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f0914ca5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0910072e30 0x7f09100cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f0914ca5640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f09100b5230 0x7f09101f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f0914ca5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0910158770 0x7f09100c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f0914ca5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0910071fa0 con 0x7f0910158770 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f0914ca5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f09100b4a30 con 0x7f0910072e30 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f0914ca5640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0910158b50 con 0x7f09100b5230 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f090ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0910158770 0x7f09100c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f090ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0910158770 0x7f09100c96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46186/0 (socket says 172.21.15.12:46186) 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f090ed9d640 1 -- 172.21.15.12:0/3577281629 learned_addr learned my addr 172.21.15.12:0/3577281629 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f090e59c640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0910072e30 0x7f09100cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.494+0000 7f0907fff640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f09100b5230 0x7f09101f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f090ed9d640 1 -- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f09100b5230 msgr2=0x7f09101f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f090ed9d640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f09100b5230 0x7f09101f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f090ed9d640 1 -- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0910072e30 msgr2=0x7f09100cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f090ed9d640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0910072e30 0x7f09100cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f090ed9d640 1 -- 172.21.15.12:0/3577281629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09100cb830 con 0x7f0910158770 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f0907fff640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f09100b5230 0x7f09101f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:28.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f090e59c640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0910072e30 0x7f09100cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:28.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f090ed9d640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0910158770 0x7f09100c96a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f090800efc0 tx=0x7f090800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f0906ffd640 1 -- 172.21.15.12:0/3577281629 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0908018030 con 0x7f0910158770 2024-04-15T15:23:28.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f0906ffd640 1 -- 172.21.15.12:0/3577281629 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f090800cdf0 con 0x7f0910158770 2024-04-15T15:23:28.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09100cbb10 con 0x7f0910158770 2024-04-15T15:23:28.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f0906ffd640 1 -- 172.21.15.12:0/3577281629 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0908003bb0 con 0x7f0910158770 2024-04-15T15:23:28.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.495+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0910149170 con 0x7f0910158770 2024-04-15T15:23:28.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.496+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0910149370 con 0x7f0910158770 2024-04-15T15:23:28.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.497+0000 7f0906ffd640 1 -- 172.21.15.12:0/3577281629 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f090801b070 con 0x7f0910158770 2024-04-15T15:23:28.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.497+0000 7f0906ffd640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f08f006c980 0x7f08f006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.497+0000 7f0906ffd640 1 -- 172.21.15.12:0/3577281629 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f090808c870 con 0x7f0910158770 2024-04-15T15:23:28.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.497+0000 7f090e59c640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f08f006c980 0x7f08f006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.498+0000 7f090e59c640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f08f006c980 0x7f08f006ee40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0900036000 tx=0x7f0900008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.502+0000 7f0906ffd640 1 -- 172.21.15.12:0/3577281629 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0910149370 con 0x7f0910158770 2024-04-15T15:23:28.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.691+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09100689e0 con 0x7f0910158770 2024-04-15T15:23:28.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.692+0000 7f0906ffd640 1 -- 172.21.15.12:0/3577281629 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17084 (secure 0 0 0) 0x7f0908010040 con 0x7f0910158770 2024-04-15T15:23:28.694 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:28.695 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":42,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:23:15.001862+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6837/3388147599":"2024-04-16T15:23:15.001839+0000","172.21.15.55:6835/3388147599":"2024-04-16T15:23:15.001839+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f08f006c980 msgr2=0x7f08f006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f08f006c980 0x7f08f006ee40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0900036000 tx=0x7f0900008030 comp rx=0 tx=0).stop 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0910158770 msgr2=0x7f09100c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0910158770 0x7f09100c96a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f090800efc0 tx=0x7f090800c490 comp rx=0 tx=0).stop 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 shutdown_connections 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0910158770 0x7f09100c96a0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f09100b5230 0x7f09101f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f08f006c980 0x7f08f006ee40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 --2- 172.21.15.12:0/3577281629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0910072e30 0x7f09100cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 >> 172.21.15.12:0/3577281629 conn(0x7f091006be00 msgr2=0x7f0910156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:28.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 shutdown_connections 2024-04-15T15:23:28.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.695+0000 7f0914ca5640 1 -- 172.21.15.12:0/3577281629 wait complete. 2024-04-15T15:23:28.708 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:20:58.603295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:22:31.098808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:22:31.597572+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:23:28.708 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:23:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93bfdcd640 1 Processor -- start 2024-04-15T15:23:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93bfdcd640 1 -- start start 2024-04-15T15:23:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93bfdcd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d15e0 0x7f93b80d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93bfdcd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93b80d4a40 0x7f93b80d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93bfdcd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93b815eb70 con 0x7f93b815e790 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93bfdcd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f93b80d3790 con 0x7f93b80d4a40 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93bfdcd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f93b80d45b0 con 0x7f93b80d15e0 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93be36a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93b80d4a40 0x7f93b80d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93be36a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93b80d4a40 0x7f93b80d4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46196/0 (socket says 172.21.15.12:46196) 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93be36a640 1 -- 172.21.15.12:0/3071392198 learned_addr learned my addr 172.21.15.12:0/3071392198 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93bd368640 1 --2- 172.21.15.12:0/3071392198 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d15e0 0x7f93b80d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93be36a640 1 -- 172.21.15.12:0/3071392198 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d15e0 msgr2=0x7f93b80d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93be36a640 1 --2- 172.21.15.12:0/3071392198 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d15e0 0x7f93b80d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93be36a640 1 -- 172.21.15.12:0/3071392198 >> v1:172.21.15.55:6789/0 conn(0x7f93b815e790 legacy=0x7f93b80b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.815+0000 7f93be36a640 1 -- 172.21.15.12:0/3071392198 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b80d3c80 con 0x7f93b80d4a40 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.816+0000 7f93be36a640 1 --2- 172.21.15.12:0/3071392198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d4a40 0x7f93b80d4eb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f93b80d3910 tx=0x7f93b002bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7940d3bd3b002e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.816+0000 7f93bcb67640 1 -- 172.21.15.12:0/3071392198 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b002c910 con 0x7f93b80d4a40 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.816+0000 7f93bcb67640 1 -- 172.21.15.12:0/3071392198 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b002cab0 con 0x7f93b80d4a40 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.816+0000 7f93bcb67640 1 -- 172.21.15.12:0/3071392198 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00343f0 con 0x7f93b80d4a40 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.817+0000 7f93bfdcd640 1 -- 172.21.15.12:0/3071392198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d4a40 msgr2=0x7f93b80d4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.817+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/3071392198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d4a40 0x7f93b80d4eb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f93b80d3910 tx=0x7f93b002bd70 comp rx=0 tx=0).stop 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.817+0000 7f93bfdcd640 1 -- 172.21.15.12:0/3071392198 shutdown_connections 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.817+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/3071392198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d4a40 0x7f93b80d4eb0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.817+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/3071392198 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d15e0 0x7f93b80d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.817+0000 7f93bfdcd640 1 -- 172.21.15.12:0/3071392198 >> 172.21.15.12:0/3071392198 conn(0x7f93b8067b40 msgr2=0x7f93b8067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:28.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.817+0000 7f93bfdcd640 1 -- 172.21.15.12:0/3071392198 shutdown_connections 2024-04-15T15:23:28.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.817+0000 7f93bfdcd640 1 -- 172.21.15.12:0/3071392198 wait complete. 2024-04-15T15:23:28.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.818+0000 7f93bfdcd640 1 Processor -- start 2024-04-15T15:23:28.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.818+0000 7f93bfdcd640 1 -- start start 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.818+0000 7f93bfdcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d15e0 0x7f93b8160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.818+0000 7f93bfdcd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d4a40 0x7f93b815d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bfdcd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b815e790 0x7f93b80cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bfdcd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80d4240 con 0x7f93b80d15e0 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bfdcd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80d53f0 con 0x7f93b815e790 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bfdcd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f93b80b7db0 con 0x7f93b80d4a40 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d15e0 0x7f93b8160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d15e0 0x7f93b8160350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46210/0 (socket says 172.21.15.12:46210) 2024-04-15T15:23:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 -- 172.21.15.12:0/145736907 learned_addr learned my addr 172.21.15.12:0/145736907 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93be36a640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b815e790 0x7f93b80cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bd368640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d4a40 0x7f93b815d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 -- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d4a40 msgr2=0x7f93b815d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d4a40 0x7f93b815d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 -- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b815e790 msgr2=0x7f93b80cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b815e790 0x7f93b80cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 -- 172.21.15.12:0/145736907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b8160ba0 con 0x7f93b80d15e0 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bd368640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d4a40 0x7f93b815d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93be36a640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b815e790 0x7f93b80cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bdb69640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d15e0 0x7f93b8160350 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f93b400c430 tx=0x7f93b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93ae7fc640 1 -- 172.21.15.12:0/145736907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b4011790 con 0x7f93b80d15e0 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93ae7fc640 1 -- 172.21.15.12:0/145736907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b4017070 con 0x7f93b80d15e0 2024-04-15T15:23:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b8160e80 con 0x7f93b80d15e0 2024-04-15T15:23:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93ae7fc640 1 -- 172.21.15.12:0/145736907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b4011bb0 con 0x7f93b80d15e0 2024-04-15T15:23:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.819+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b8161440 con 0x7f93b80d15e0 2024-04-15T15:23:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.820+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b80689e0 con 0x7f93b80d15e0 2024-04-15T15:23:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.821+0000 7f93ae7fc640 1 -- 172.21.15.12:0/145736907 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f93b401c460 con 0x7f93b80d15e0 2024-04-15T15:23:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.821+0000 7f93ae7fc640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f939c06c980 0x7f939c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.821+0000 7f93ae7fc640 1 -- 172.21.15.12:0/145736907 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f93b408bc10 con 0x7f93b80d15e0 2024-04-15T15:23:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.821+0000 7f93bd368640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f939c06c980 0x7f939c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.822+0000 7f93bd368640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f939c06c980 0x7f939c06ee40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f93a8006fd0 tx=0x7f93a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:28.826+0000 7f93ae7fc640 1 -- 172.21.15.12:0/145736907 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f93b4059ee0 con 0x7f93b80d15e0 2024-04-15T15:23:29.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.008+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f93b8160890 con 0x7f93b80d15e0 2024-04-15T15:23:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.008+0000 7f93ae7fc640 1 -- 172.21.15.12:0/145736907 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f93b40597e0 con 0x7f93b80d15e0 2024-04-15T15:23:29.011 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f939c06c980 msgr2=0x7f939c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f939c06c980 0x7f939c06ee40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f93a8006fd0 tx=0x7f93a8008040 comp rx=0 tx=0).stop 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d15e0 msgr2=0x7f93b8160350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d15e0 0x7f93b8160350 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f93b400c430 tx=0x7f93b400c8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 shutdown_connections 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f939c06c980 0x7f939c06ee40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b815e790 0x7f93b80cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f93b80d4a40 0x7f93b815d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 --2- 172.21.15.12:0/145736907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93b80d15e0 0x7f93b8160350 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 >> 172.21.15.12:0/145736907 conn(0x7f93b8067b40 msgr2=0x7f93b80b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:29.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 shutdown_connections 2024-04-15T15:23:29.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.011+0000 7f93bfdcd640 1 -- 172.21.15.12:0/145736907 wait complete. 2024-04-15T15:23:29.024 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:23:29.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.130+0000 7f494b20d640 1 Processor -- start 2024-04-15T15:23:29.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.131+0000 7f494b20d640 1 -- start start 2024-04-15T15:23:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.131+0000 7f494b20d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f494415e700 0x7f49440b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.131+0000 7f494b20d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f49440d2360 0x7f49440d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.131+0000 7f494b20d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49440b47a0 con 0x7f494415eae0 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.131+0000 7f494b20d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f49440b53b0 con 0x7f494415e700 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.131+0000 7f494b20d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f49440b4a80 con 0x7f49440d2360 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.131+0000 7f4948fa9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f494415e700 0x7f49440b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.132+0000 7f4948fa9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f494415e700 0x7f49440b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46214/0 (socket says 172.21.15.12:46214) 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.132+0000 7f4948fa9640 1 -- 172.21.15.12:0/374290698 learned_addr learned my addr 172.21.15.12:0/374290698 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.132+0000 7f4943fff640 1 --2- 172.21.15.12:0/374290698 >> v2:172.21.15.55:3300/0 conn(0x7f49440d2360 0x7f49440d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.132+0000 7f4948fa9640 1 -- 172.21.15.12:0/374290698 >> v2:172.21.15.55:3300/0 conn(0x7f49440d2360 msgr2=0x7f49440d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.132+0000 7f4948fa9640 1 --2- 172.21.15.12:0/374290698 >> v2:172.21.15.55:3300/0 conn(0x7f49440d2360 0x7f49440d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.132+0000 7f4948fa9640 1 -- 172.21.15.12:0/374290698 >> v1:172.21.15.55:6789/0 conn(0x7f494415eae0 legacy=0x7f49440cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.132+0000 7f4948fa9640 1 -- 172.21.15.12:0/374290698 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49440b4d60 con 0x7f494415e700 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f4948fa9640 1 --2- 172.21.15.12:0/374290698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f494415e700 0x7f49440b5840 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f494408c7b0 tx=0x7f493c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79683f9df0b20273 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f49437fe640 1 -- 172.21.15.12:0/374290698 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c02d040 con 0x7f494415e700 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f49437fe640 1 -- 172.21.15.12:0/374290698 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f493c004230 con 0x7f494415e700 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f49437fe640 1 -- 172.21.15.12:0/374290698 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c003610 con 0x7f494415e700 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f494b20d640 1 -- 172.21.15.12:0/374290698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f494415e700 msgr2=0x7f49440b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f494b20d640 1 --2- 172.21.15.12:0/374290698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f494415e700 0x7f49440b5840 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f494408c7b0 tx=0x7f493c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f494b20d640 1 -- 172.21.15.12:0/374290698 shutdown_connections 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f494b20d640 1 --2- 172.21.15.12:0/374290698 >> v2:172.21.15.55:3300/0 conn(0x7f49440d2360 0x7f49440d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f494b20d640 1 --2- 172.21.15.12:0/374290698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f494415e700 0x7f49440b5840 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f494b20d640 1 -- 172.21.15.12:0/374290698 >> 172.21.15.12:0/374290698 conn(0x7f494408bf60 msgr2=0x7f49440d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f494b20d640 1 -- 172.21.15.12:0/374290698 shutdown_connections 2024-04-15T15:23:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.133+0000 7f494b20d640 1 -- 172.21.15.12:0/374290698 wait complete. 2024-04-15T15:23:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.134+0000 7f494b20d640 1 Processor -- start 2024-04-15T15:23:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.135+0000 7f494b20d640 1 -- start start 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.135+0000 7f494b20d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49440d2360 0x7f49441f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.135+0000 7f494b20d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f494415e700 0x7f49440b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.135+0000 7f494b20d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f494415eae0 0x7f494406dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.135+0000 7f494b20d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f49440b5d80 con 0x7f49440d2360 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.135+0000 7f494b20d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f49440b4450 con 0x7f494415eae0 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.135+0000 7f494b20d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f49440d1d40 con 0x7f494415e700 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.135+0000 7f4948fa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49440d2360 0x7f49441f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f4948fa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49440d2360 0x7f49441f74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46226/0 (socket says 172.21.15.12:46226) 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f4948fa9640 1 -- 172.21.15.12:0/1361557656 learned_addr learned my addr 172.21.15.12:0/1361557656 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f4948fa9640 1 -- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f494415e700 msgr2=0x7f49440b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f4948fa9640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f494415e700 0x7f49440b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f4948fa9640 1 -- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f494415eae0 msgr2=0x7f494406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f4948fa9640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f494415eae0 0x7f494406dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f4948fa9640 1 -- 172.21.15.12:0/1361557656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49440c8330 con 0x7f49440d2360 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f4948fa9640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49440d2360 0x7f49441f74c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f493c02b930 tx=0x7f493c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f49417fa640 1 -- 172.21.15.12:0/1361557656 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c003e50 con 0x7f49440d2360 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f49417fa640 1 -- 172.21.15.12:0/1361557656 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f493c006040 con 0x7f49440d2360 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49440c8610 con 0x7f49440d2360 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f49417fa640 1 -- 172.21.15.12:0/1361557656 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c006430 con 0x7f49440d2360 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.136+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49440c8bd0 con 0x7f49440d2360 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.137+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49440c9000 con 0x7f49440d2360 2024-04-15T15:23:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.138+0000 7f49417fa640 1 -- 172.21.15.12:0/1361557656 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f493c0065d0 con 0x7f49440d2360 2024-04-15T15:23:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.138+0000 7f49417fa640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f492406cb50 0x7f492406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.138+0000 7f49417fa640 1 -- 172.21.15.12:0/1361557656 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f493c0ae160 con 0x7f49440d2360 2024-04-15T15:23:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.138+0000 7f4943fff640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f492406cb50 0x7f492406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.139+0000 7f4943fff640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f492406cb50 0x7f492406f010 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f492c006fd0 tx=0x7f492c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.143+0000 7f49417fa640 1 -- 172.21.15.12:0/1361557656 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f493c07c290 con 0x7f49440d2360 2024-04-15T15:23:29.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.328+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f49440689e0 con 0x7f49440d2360 2024-04-15T15:23:29.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.329+0000 7f49417fa640 1 -- 172.21.15.12:0/1361557656 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f493c07bb90 con 0x7f49440d2360 2024-04-15T15:23:29.330 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f492406cb50 msgr2=0x7f492406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f492406cb50 0x7f492406f010 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f492c006fd0 tx=0x7f492c008040 comp rx=0 tx=0).stop 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49440d2360 msgr2=0x7f49441f74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49440d2360 0x7f49441f74c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f493c02b930 tx=0x7f493c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 shutdown_connections 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f492406cb50 0x7f492406f010 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f494415eae0 0x7f494406dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f494415e700 0x7f49440b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 --2- 172.21.15.12:0/1361557656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f49440d2360 0x7f49441f74c0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 >> 172.21.15.12:0/1361557656 conn(0x7f494408bf60 msgr2=0x7f494415ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 shutdown_connections 2024-04-15T15:23:29.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.331+0000 7f494b20d640 1 -- 172.21.15.12:0/1361557656 wait complete. 2024-04-15T15:23:29.343 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:23:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.450+0000 7f7f23014640 1 Processor -- start 2024-04-15T15:23:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.450+0000 7f7f23014640 1 -- start start 2024-04-15T15:23:29.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.450+0000 7f7f23014640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f1c09b880 0x7f7f1c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.450+0000 7f7f23014640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f1c0d05e0 con 0x7f7f1c070940 2024-04-15T15:23:29.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.450+0000 7f7f23014640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f1c071260 con 0x7f7f1c0d0da0 2024-04-15T15:23:29.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.450+0000 7f7f23014640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7f1c09bce0 con 0x7f7f1c09b880 2024-04-15T15:23:29.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.451+0000 7f7f1bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f1c09b880 0x7f7f1c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.451+0000 7f7f1bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f1c09b880 0x7f7f1c070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46230/0 (socket says 172.21.15.12:46230) 2024-04-15T15:23:29.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.451+0000 7f7f1bfff640 1 -- 172.21.15.12:0/2924670024 learned_addr learned my addr 172.21.15.12:0/2924670024 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:29.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.451+0000 7f7f1bfff640 1 -- 172.21.15.12:0/2924670024 >> v1:172.21.15.55:6789/0 conn(0x7f7f1c0d0da0 legacy=0x7f7f1c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.451+0000 7f7f1bfff640 1 -- 172.21.15.12:0/2924670024 >> v1:172.21.15.12:6789/0 conn(0x7f7f1c070940 legacy=0x7f7f1c160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.451+0000 7f7f1bfff640 1 -- 172.21.15.12:0/2924670024 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c15f720 con 0x7f7f1c09b880 2024-04-15T15:23:29.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f1bfff640 1 --2- 172.21.15.12:0/2924670024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c09b880 0x7f7f1c070d20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009870 tx=0x7f7f0802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebc3c455543bd04 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f1b7fe640 1 -- 172.21.15.12:0/2924670024 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0802f840 con 0x7f7f1c09b880 2024-04-15T15:23:29.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f1b7fe640 1 -- 172.21.15.12:0/2924670024 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0802f9e0 con 0x7f7f1c09b880 2024-04-15T15:23:29.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f1b7fe640 1 -- 172.21.15.12:0/2924670024 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0802fdb0 con 0x7f7f1c09b880 2024-04-15T15:23:29.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f23014640 1 -- 172.21.15.12:0/2924670024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c09b880 msgr2=0x7f7f1c070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f23014640 1 --2- 172.21.15.12:0/2924670024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c09b880 0x7f7f1c070d20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009870 tx=0x7f7f0802ba20 comp rx=0 tx=0).stop 2024-04-15T15:23:29.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f23014640 1 -- 172.21.15.12:0/2924670024 shutdown_connections 2024-04-15T15:23:29.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f23014640 1 --2- 172.21.15.12:0/2924670024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c09b880 0x7f7f1c070d20 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f23014640 1 -- 172.21.15.12:0/2924670024 >> 172.21.15.12:0/2924670024 conn(0x7f7f1c0b3f40 msgr2=0x7f7f1c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:29.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f23014640 1 -- 172.21.15.12:0/2924670024 shutdown_connections 2024-04-15T15:23:29.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.452+0000 7f7f23014640 1 -- 172.21.15.12:0/2924670024 wait complete. 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.453+0000 7f7f23014640 1 Processor -- start 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.454+0000 7f7f23014640 1 -- start start 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.454+0000 7f7f23014640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c070940 0x7f7f1c0b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.454+0000 7f7f23014640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f1c09b880 0x7f7f1c0b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.454+0000 7f7f23014640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f1c0d0da0 0x7f7f1c14d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.454+0000 7f7f23014640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c0d0880 con 0x7f7f1c070940 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.454+0000 7f7f23014640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c15fce0 con 0x7f7f1c0d0da0 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.454+0000 7f7f23014640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7f1c0d18e0 con 0x7f7f1c09b880 2024-04-15T15:23:29.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c070940 0x7f7f1c0b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c070940 0x7f7f1c0b5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46242/0 (socket says 172.21.15.12:46242) 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 -- 172.21.15.12:0/1083162534 learned_addr learned my addr 172.21.15.12:0/1083162534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f215b1640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f1c0d0da0 0x7f7f1c14d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 -- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f1c09b880 msgr2=0x7f7f1c0b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f1c09b880 0x7f7f1c0b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 -- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f1c0d0da0 msgr2=0x7f7f1c14d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f1c0d0da0 0x7f7f1c14d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 -- 172.21.15.12:0/1083162534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c0b6500 con 0x7f7f1c070940 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f215b1640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f1c0d0da0 0x7f7f1c14d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f20db0640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c070940 0x7f7f1c0b5cb0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000c400 tx=0x7f7f1000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f197fa640 1 -- 172.21.15.12:0/1083162534 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10017070 con 0x7f7f1c070940 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f197fa640 1 -- 172.21.15.12:0/1083162534 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f100052d0 con 0x7f7f1c070940 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1c0b67e0 con 0x7f7f1c070940 2024-04-15T15:23:29.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f197fa640 1 -- 172.21.15.12:0/1083162534 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10003830 con 0x7f7f1c070940 2024-04-15T15:23:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.455+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f1c1e6f50 con 0x7f7f1c070940 2024-04-15T15:23:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.456+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f1c0b6d00 con 0x7f7f1c070940 2024-04-15T15:23:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.457+0000 7f7f197fa640 1 -- 172.21.15.12:0/1083162534 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7f1001a070 con 0x7f7f1c070940 2024-04-15T15:23:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.457+0000 7f7f197fa640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7f0006c9b0 0x7f7f0006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.457+0000 7f7f197fa640 1 -- 172.21.15.12:0/1083162534 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f7f1008b080 con 0x7f7f1c070940 2024-04-15T15:23:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.457+0000 7f7f1bfff640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7f0006c9b0 0x7f7f0006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.458+0000 7f7f1bfff640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7f0006c9b0 0x7f7f0006ee70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7f08036000 tx=0x7f7f08039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.462+0000 7f7f197fa640 1 -- 172.21.15.12:0/1083162534 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f1c0b6d00 con 0x7f7f1c070940 2024-04-15T15:23:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.647+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7f1c0689e0 con 0x7f7f1c070940 2024-04-15T15:23:29.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.647+0000 7f7f197fa640 1 -- 172.21.15.12:0/1083162534 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f7f100592d0 con 0x7f7f1c070940 2024-04-15T15:23:29.649 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7f0006c9b0 msgr2=0x7f7f0006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7f0006c9b0 0x7f7f0006ee70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7f08036000 tx=0x7f7f08039040 comp rx=0 tx=0).stop 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c070940 msgr2=0x7f7f1c0b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c070940 0x7f7f1c0b5cb0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000c400 tx=0x7f7f1000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 shutdown_connections 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7f0006c9b0 0x7f7f0006ee70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f1c0d0da0 0x7f7f1c14d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7f1c09b880 0x7f7f1c0b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 --2- 172.21.15.12:0/1083162534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f1c070940 0x7f7f1c0b5cb0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 >> 172.21.15.12:0/1083162534 conn(0x7f7f1c0b3f40 msgr2=0x7f7f1c161880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 shutdown_connections 2024-04-15T15:23:29.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.649+0000 7f7f23014640 1 -- 172.21.15.12:0/1083162534 wait complete. 2024-04-15T15:23:29.662 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:23:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.764+0000 7f6ded3cd640 1 Processor -- start 2024-04-15T15:23:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.764+0000 7f6ded3cd640 1 -- start start 2024-04-15T15:23:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.765+0000 7f6ded3cd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6de80d2360 0x7f6de80d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.765+0000 7f6ded3cd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de80b47a0 con 0x7f6de815eae0 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.765+0000 7f6ded3cd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de80b4a80 con 0x7f6de815e700 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.765+0000 7f6ded3cd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6de80b53b0 con 0x7f6de80d2360 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.765+0000 7f6de77fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6de815eae0 0x7f6de80cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47114/0 (socket says 172.21.15.12:47114) 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.765+0000 7f6de77fe640 1 -- 172.21.15.12:0/2600463565 learned_addr learned my addr 172.21.15.12:0/2600463565 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de67fc640 1 --2- 172.21.15.12:0/2600463565 >> v2:172.21.15.55:3300/0 conn(0x7f6de80d2360 0x7f6de80d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747638173 0 0) 0x7f6de80b47a0 con 0x7f6de815eae0 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd4003520 con 0x7f6de815eae0 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2032550424 0 0) 0x7f6dd4003520 con 0x7f6de815eae0 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 >> v2:172.21.15.55:3300/0 conn(0x7f6de80d2360 msgr2=0x7f6de80d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 --2- 172.21.15.12:0/2600463565 >> v2:172.21.15.55:3300/0 conn(0x7f6de80d2360 0x7f6de80d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 >> v1:172.21.15.55:6789/0 conn(0x7f6de815e700 legacy=0x7f6de80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de80b4d60 con 0x7f6de815eae0 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f6dd8001670 con 0x7f6de815eae0 2024-04-15T15:23:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 >> v1:172.21.15.12:6789/0 conn(0x7f6de815eae0 legacy=0x7f6de80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6dd4003a00 0x7f6dd4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dd4004800 0x7f6dd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd40051c0 0x7f6dd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6de80b53b0 con 0x7f6dd4004800 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd400c0f0 con 0x7f6dd40051c0 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd400c2f0 con 0x7f6dd4003a00 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de67fc640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dd4004800 0x7f6dd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de6ffd640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6dd4003a00 0x7f6dd4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de77fe640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd40051c0 0x7f6dd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de67fc640 1 -- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6dd4003a00 msgr2=0x7f6dd4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de67fc640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6dd4003a00 0x7f6dd4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de67fc640 1 -- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd40051c0 msgr2=0x7f6dd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de67fc640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd40051c0 0x7f6dd400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de67fc640 1 -- 172.21.15.12:0/2600463565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd4003520 con 0x7f6dd4004800 2024-04-15T15:23:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de77fe640 1 -- 172.21.15.12:0/2600463565 reap_dead start 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.766+0000 7f6de6ffd640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6dd4003a00 0x7f6dd4003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.767+0000 7f6de67fc640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dd4004800 0x7f6dd4004c80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009ac0 tx=0x7f6dd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58488b03241c5aab server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.767+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0039070 con 0x7f6dd4004800 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.767+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd002c6b0 con 0x7f6dd4004800 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.767+0000 7f6de5ffb640 1 -- 172.21.15.12:0/2600463565 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0009690 con 0x7f6dd4004800 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.768+0000 7f6ded3cd640 1 -- 172.21.15.12:0/2600463565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dd4004800 msgr2=0x7f6dd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.768+0000 7f6ded3cd640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dd4004800 0x7f6dd4004c80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009ac0 tx=0x7f6dd4005830 comp rx=0 tx=0).stop 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.768+0000 7f6ded3cd640 1 -- 172.21.15.12:0/2600463565 shutdown_connections 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.768+0000 7f6ded3cd640 1 --2- 172.21.15.12:0/2600463565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dd4004800 0x7f6dd4004c80 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.768+0000 7f6ded3cd640 1 -- 172.21.15.12:0/2600463565 >> 172.21.15.12:0/2600463565 conn(0x7f6de808bf60 msgr2=0x7f6de80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.768+0000 7f6ded3cd640 1 -- 172.21.15.12:0/2600463565 shutdown_connections 2024-04-15T15:23:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.768+0000 7f6ded3cd640 1 -- 172.21.15.12:0/2600463565 wait complete. 2024-04-15T15:23:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.769+0000 7f6ded3cd640 1 Processor -- start 2024-04-15T15:23:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.769+0000 7f6ded3cd640 1 -- start start 2024-04-15T15:23:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6ded3cd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd4004800 0x7f6de80ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6ded3cd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6de80ca770 0x7f6de80c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6ded3cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de80c8080 0x7f6de8160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6ded3cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6de80d1d40 con 0x7f6de80c8080 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6ded3cd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6de80b4450 con 0x7f6dd4004800 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6ded3cd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6de80d1ec0 con 0x7f6de80ca770 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de80c8080 0x7f6de8160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de80c8080 0x7f6de8160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46258/0 (socket says 172.21.15.12:46258) 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 -- 172.21.15.12:0/123071481 learned_addr learned my addr 172.21.15.12:0/123071481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de67fc640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6de80ca770 0x7f6de80c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de6ffd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd4004800 0x7f6de80ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 -- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6de80ca770 msgr2=0x7f6de80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6de80ca770 0x7f6de80c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 -- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd4004800 msgr2=0x7f6de80ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd4004800 0x7f6de80ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 -- 172.21.15.12:0/123071481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de8160ce0 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de67fc640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6de80ca770 0x7f6de80c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de6ffd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd4004800 0x7f6de80ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.770+0000 7f6de77fe640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de80c8080 0x7f6de8160020 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8000fb0 tx=0x7f6dd800e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.771+0000 7f6ddf7fe640 1 -- 172.21.15.12:0/123071481 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8019070 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.771+0000 7f6ddf7fe640 1 -- 172.21.15.12:0/123071481 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd8010510 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.771+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de8160fc0 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.771+0000 7f6ddf7fe640 1 -- 172.21.15.12:0/123071481 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd80108e0 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.771+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de8161580 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.772+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de80689e0 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.772+0000 7f6ddf7fe640 1 -- 172.21.15.12:0/123071481 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6dd801c070 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.773+0000 7f6ddf7fe640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6dc806c980 0x7f6dc806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.773+0000 7f6ddf7fe640 1 -- 172.21.15.12:0/123071481 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f6dd808b430 con 0x7f6de80c8080 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.773+0000 7f6de6ffd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6dc806c980 0x7f6dc806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.773+0000 7f6de6ffd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6dc806c980 0x7f6dc806ee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6de806a8e0 tx=0x7f6de0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.777+0000 7f6ddf7fe640 1 -- 172.21.15.12:0/123071481 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6dd80595e0 con 0x7f6de80c8080 2024-04-15T15:23:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.999+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6de80ba500 con 0x7f6de80c8080 2024-04-15T15:23:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:29.999+0000 7f6ddf7fe640 1 -- 172.21.15.12:0/123071481 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5180 (secure 0 0 0) 0x7f6dd8058ee0 con 0x7f6de80c8080 2024-04-15T15:23:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:30.001 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5102,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/852515692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":852515692},{"type":"v1","addr":"172.21.15.55:6837","nonce":852515692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:23:19.042212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5200":{"gid":5200,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5200,"qdb_cluster":[5200]},"id":2}]} 2024-04-15T15:23:30.001 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-15T15:23:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6dc806c980 msgr2=0x7f6dc806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6dc806c980 0x7f6dc806ee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6de806a8e0 tx=0x7f6de0009040 comp rx=0 tx=0).stop 2024-04-15T15:23:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de80c8080 msgr2=0x7f6de8160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de80c8080 0x7f6de8160020 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8000fb0 tx=0x7f6dd800e910 comp rx=0 tx=0).stop 2024-04-15T15:23:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 shutdown_connections 2024-04-15T15:23:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de80c8080 0x7f6de8160020 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6de80ca770 0x7f6de80c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6dc806c980 0x7f6dc806ee40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 --2- 172.21.15.12:0/123071481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd4004800 0x7f6de80ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 >> 172.21.15.12:0/123071481 conn(0x7f6de808bf60 msgr2=0x7f6de80d0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.001+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 shutdown_connections 2024-04-15T15:23:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.002+0000 7f6ded3cd640 1 -- 172.21.15.12:0/123071481 wait complete. 2024-04-15T15:23:30.015 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:23:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.118+0000 7f879c293640 1 Processor -- start 2024-04-15T15:23:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.118+0000 7f879c293640 1 -- start start 2024-04-15T15:23:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879c293640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87940d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879c293640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f87940d1610 0x7f87940d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879c293640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87940d2480 con 0x7f87940cfd20 2024-04-15T15:23:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879c293640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f87940d2db0 con 0x7f87940d1610 2024-04-15T15:23:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879c293640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f87940d2760 con 0x7f879415e790 2024-04-15T15:23:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879a02f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87940d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879a830640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f87940d1610 0x7f87940d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879a02f640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87940d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:47036/0 (socket says 172.21.15.12:47036) 2024-04-15T15:23:30.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879a02f640 1 -- 172.21.15.12:0/1486044292 learned_addr learned my addr 172.21.15.12:0/1486044292 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:30.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.119+0000 7f879982e640 1 --1- 172.21.15.12:0/1486044292 >> v1:172.21.15.55:6789/0 conn(0x7f87940cfd20 0x7f87940d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:42836/0 (socket says 172.21.15.12:42836) 2024-04-15T15:23:30.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.120+0000 7f879a02f640 1 -- 172.21.15.12:0/1486044292 >> v2:172.21.15.55:3300/0 conn(0x7f87940d1610 msgr2=0x7f87940d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:30.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.120+0000 7f879a02f640 1 --2- 172.21.15.12:0/1486044292 >> v2:172.21.15.55:3300/0 conn(0x7f87940d1610 0x7f87940d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.120+0000 7f879a02f640 1 -- 172.21.15.12:0/1486044292 >> v1:172.21.15.55:6789/0 conn(0x7f87940cfd20 legacy=0x7f87940d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.120+0000 7f879a02f640 1 -- 172.21.15.12:0/1486044292 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87940d2a40 con 0x7f879415e790 2024-04-15T15:23:30.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.120+0000 7f879a830640 1 --2- 172.21.15.12:0/1486044292 >> v2:172.21.15.55:3300/0 conn(0x7f87940d1610 0x7f87940d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:30.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.120+0000 7f879a02f640 1 --2- 172.21.15.12:0/1486044292 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87940d3240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8790009870 tx=0x7f879002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c2ca0ac7b201c82 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879902d640 1 -- 172.21.15.12:0/1486044292 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790034070 con 0x7f879415e790 2024-04-15T15:23:30.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879902d640 1 -- 172.21.15.12:0/1486044292 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8790030840 con 0x7f879415e790 2024-04-15T15:23:30.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879902d640 1 -- 172.21.15.12:0/1486044292 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790030c10 con 0x7f879415e790 2024-04-15T15:23:30.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879c293640 1 -- 172.21.15.12:0/1486044292 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 msgr2=0x7f87940d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879c293640 1 --2- 172.21.15.12:0/1486044292 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87940d3240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8790009870 tx=0x7f879002b980 comp rx=0 tx=0).stop 2024-04-15T15:23:30.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879c293640 1 -- 172.21.15.12:0/1486044292 shutdown_connections 2024-04-15T15:23:30.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879c293640 1 --2- 172.21.15.12:0/1486044292 >> v2:172.21.15.55:3300/0 conn(0x7f87940d1610 0x7f87940d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879c293640 1 --2- 172.21.15.12:0/1486044292 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87940d3240 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879c293640 1 -- 172.21.15.12:0/1486044292 >> 172.21.15.12:0/1486044292 conn(0x7f879408bf60 msgr2=0x7f87940d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:30.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879c293640 1 -- 172.21.15.12:0/1486044292 shutdown_connections 2024-04-15T15:23:30.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.121+0000 7f879c293640 1 -- 172.21.15.12:0/1486044292 wait complete. 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.122+0000 7f879c293640 1 Processor -- start 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879c293640 1 -- start start 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879c293640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87940cfd20 0x7f87940cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879c293640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87940d1610 0x7f8794145b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879c293640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87941460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879c293640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f879415ecf0 con 0x7f87940cfd20 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879c293640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f879415eb70 con 0x7f87940d1610 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879c293640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f87940b74f0 con 0x7f879415e790 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879a02f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87940cfd20 0x7f87940cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879a02f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87940cfd20 0x7f87940cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46262/0 (socket says 172.21.15.12:46262) 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.123+0000 7f879a02f640 1 -- 172.21.15.12:0/3165982589 learned_addr learned my addr 172.21.15.12:0/3165982589 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879a02f640 1 -- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 msgr2=0x7f87941460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879a02f640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87941460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879a02f640 1 -- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87940d1610 msgr2=0x7f8794145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879982e640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87940d1610 0x7f8794145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879a02f640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87940d1610 0x7f8794145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879a02f640 1 -- 172.21.15.12:0/3165982589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8794143c10 con 0x7f87940cfd20 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879982e640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87940d1610 0x7f8794145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879a02f640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87940cfd20 0x7f87940cac30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c00 tx=0x7f87900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f878affd640 1 -- 172.21.15.12:0/3165982589 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790034040 con 0x7f87940cfd20 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f878affd640 1 -- 172.21.15.12:0/3165982589 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8790039d10 con 0x7f87940cfd20 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8794143ef0 con 0x7f87940cfd20 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f878affd640 1 -- 172.21.15.12:0/3165982589 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790043630 con 0x7f87940cfd20 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.124+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87940cb170 con 0x7f87940cfd20 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.125+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87940689e0 con 0x7f87940cfd20 2024-04-15T15:23:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.126+0000 7f878affd640 1 -- 172.21.15.12:0/3165982589 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f879004a060 con 0x7f87940cfd20 2024-04-15T15:23:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.126+0000 7f878affd640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f877806c980 0x7f877806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.126+0000 7f878affd640 1 -- 172.21.15.12:0/3165982589 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f87900add20 con 0x7f87940cfd20 2024-04-15T15:23:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.126+0000 7f879982e640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f877806c980 0x7f877806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.127+0000 7f879982e640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f877806c980 0x7f877806ee40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f87940b9f10 tx=0x7f8784009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.130+0000 7f878affd640 1 -- 172.21.15.12:0/3165982589 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f879003e070 con 0x7f87940cfd20 2024-04-15T15:23:30.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.316+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87940cf200 con 0x7f87940cfd20 2024-04-15T15:23:30.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.317+0000 7f878affd640 1 -- 172.21.15.12:0/3165982589 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17084 (secure 0 0 0) 0x7f87900ad090 con 0x7f87940cfd20 2024-04-15T15:23:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:30.318 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":42,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:23:15.001862+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6837/3388147599":"2024-04-16T15:23:15.001839+0000","172.21.15.55:6835/3388147599":"2024-04-16T15:23:15.001839+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f877806c980 msgr2=0x7f877806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f877806c980 0x7f877806ee40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f87940b9f10 tx=0x7f8784009040 comp rx=0 tx=0).stop 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87940cfd20 msgr2=0x7f87940cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87940cfd20 0x7f87940cac30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c00 tx=0x7f87900046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 shutdown_connections 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f879415e790 0x7f87941460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f877806c980 0x7f877806ee40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87940d1610 0x7f8794145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 --2- 172.21.15.12:0/3165982589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87940cfd20 0x7f87940cac30 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 >> 172.21.15.12:0/3165982589 conn(0x7f879408bf60 msgr2=0x7f87940b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 shutdown_connections 2024-04-15T15:23:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.319+0000 7f879c293640 1 -- 172.21.15.12:0/3165982589 wait complete. 2024-04-15T15:23:30.332 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:23:30.332 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:23:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.435+0000 7f6f2fc41640 1 Processor -- start 2024-04-15T15:23:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.435+0000 7f6f2fc41640 1 -- start start 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2fc41640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6f28070940 0x7f6f2809b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2fc41640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f28160310 0x7f6f28070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2fc41640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f280d0da0 0x7f6f280d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2fc41640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6f280d05e0 con 0x7f6f28160310 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2fc41640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6f28071260 con 0x7f6f28070940 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2fc41640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6f2815f720 con 0x7f6f280d0da0 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d1dc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f28160310 0x7f6f28070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d1dc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f28160310 0x7f6f28070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46264/0 (socket says 172.21.15.12:46264) 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d1dc640 1 -- 172.21.15.12:0/1299065578 learned_addr learned my addr 172.21.15.12:0/1299065578 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d9dd640 1 --2- 172.21.15.12:0/1299065578 >> v2:172.21.15.55:3300/0 conn(0x7f6f28070940 0x7f6f2809b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2e1de640 1 --2- 172.21.15.12:0/1299065578 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f280d0da0 0x7f6f280d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d1dc640 1 -- 172.21.15.12:0/1299065578 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f280d0da0 msgr2=0x7f6f280d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d1dc640 1 --2- 172.21.15.12:0/1299065578 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f280d0da0 0x7f6f280d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d1dc640 1 -- 172.21.15.12:0/1299065578 >> v2:172.21.15.55:3300/0 conn(0x7f6f28070940 msgr2=0x7f6f2809b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d1dc640 1 --2- 172.21.15.12:0/1299065578 >> v2:172.21.15.55:3300/0 conn(0x7f6f28070940 0x7f6f2809b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d1dc640 1 -- 172.21.15.12:0/1299065578 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2815fa00 con 0x7f6f28160310 2024-04-15T15:23:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.436+0000 7f6f2d9dd640 1 --2- 172.21.15.12:0/1299065578 >> v2:172.21.15.55:3300/0 conn(0x7f6f28070940 0x7f6f2809b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.437+0000 7f6f2d1dc640 1 --2- 172.21.15.12:0/1299065578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f28160310 0x7f6f28070d20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6f280741c0 tx=0x7f6f1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc80da292ce5bf2b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.437+0000 7f6f2c9db640 1 -- 172.21.15.12:0/1299065578 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18035020 con 0x7f6f28160310 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.437+0000 7f6f2c9db640 1 -- 172.21.15.12:0/1299065578 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f18031870 con 0x7f6f28160310 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.437+0000 7f6f2c9db640 1 -- 172.21.15.12:0/1299065578 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18031c40 con 0x7f6f28160310 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 -- 172.21.15.12:0/1299065578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f28160310 msgr2=0x7f6f28070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/1299065578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f28160310 0x7f6f28070d20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6f280741c0 tx=0x7f6f1802b910 comp rx=0 tx=0).stop 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 -- 172.21.15.12:0/1299065578 shutdown_connections 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/1299065578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f28160310 0x7f6f28070d20 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/1299065578 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f280d0da0 0x7f6f280d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/1299065578 >> v2:172.21.15.55:3300/0 conn(0x7f6f28070940 0x7f6f2809b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 -- 172.21.15.12:0/1299065578 >> 172.21.15.12:0/1299065578 conn(0x7f6f280b3f40 msgr2=0x7f6f280b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 -- 172.21.15.12:0/1299065578 shutdown_connections 2024-04-15T15:23:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.438+0000 7f6f2fc41640 1 -- 172.21.15.12:0/1299065578 wait complete. 2024-04-15T15:23:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.439+0000 7f6f2fc41640 1 Processor -- start 2024-04-15T15:23:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.439+0000 7f6f2fc41640 1 -- start start 2024-04-15T15:23:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2fc41640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f28070940 0x7f6f28150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2fc41640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f280d0da0 0x7f6f281513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2fc41640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f28160310 0x7f6f280c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2fc41640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f28160770 con 0x7f6f280d0da0 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2fc41640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6f280d0880 con 0x7f6f28070940 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2fc41640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f6f280d1750 con 0x7f6f28160310 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f280d0da0 0x7f6f281513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f280d0da0 0x7f6f281513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46276/0 (socket says 172.21.15.12:46276) 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 -- 172.21.15.12:0/227539689 learned_addr learned my addr 172.21.15.12:0/227539689 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d9dd640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f28070940 0x7f6f28150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2e1de640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f28160310 0x7f6f280c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 -- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f28160310 msgr2=0x7f6f280c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f28160310 0x7f6f280c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 -- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f28070940 msgr2=0x7f6f28150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f28070940 0x7f6f28150820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 -- 172.21.15.12:0/227539689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f280c5f60 con 0x7f6f280d0da0 2024-04-15T15:23:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.440+0000 7f6f2d1dc640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f280d0da0 0x7f6f281513b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6f18000c00 tx=0x7f6f180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.441+0000 7f6f1e7fc640 1 -- 172.21.15.12:0/227539689 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18035040 con 0x7f6f280d0da0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.441+0000 7f6f1e7fc640 1 -- 172.21.15.12:0/227539689 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f18030c80 con 0x7f6f280d0da0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.441+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f280c6240 con 0x7f6f280d0da0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.441+0000 7f6f1e7fc640 1 -- 172.21.15.12:0/227539689 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f180456a0 con 0x7f6f280d0da0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.441+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f280c6800 con 0x7f6f280d0da0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.442+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f280689e0 con 0x7f6f280d0da0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.442+0000 7f6f1e7fc640 1 -- 172.21.15.12:0/227539689 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6f18038020 con 0x7f6f280d0da0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.443+0000 7f6f1e7fc640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f0c06c980 0x7f6f0c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.443+0000 7f6f1e7fc640 1 -- 172.21.15.12:0/227539689 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f6f180ae160 con 0x7f6f280d0da0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.443+0000 7f6f2d9dd640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f0c06c980 0x7f6f0c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.443+0000 7f6f2d9dd640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f0c06c980 0x7f6f0c06ee40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6f24006fd0 tx=0x7f6f24008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.447+0000 7f6f1e7fc640 1 -- 172.21.15.12:0/227539689 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f1807c330 con 0x7f6f280d0da0 2024-04-15T15:23:30.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.673+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f28070720 con 0x7f6f280d0da0 2024-04-15T15:23:30.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.673+0000 7f6f1e7fc640 1 -- 172.21.15.12:0/227539689 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5180 (secure 0 0 0) 0x7f6f1807bc30 con 0x7f6f280d0da0 2024-04-15T15:23:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:30.675 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5102,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/852515692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":852515692},{"type":"v1","addr":"172.21.15.55:6837","nonce":852515692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:23:19.042212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5200":{"gid":5200,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5200,"qdb_cluster":[5200]},"id":2}]} 2024-04-15T15:23:30.675 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f0c06c980 msgr2=0x7f6f0c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f0c06c980 0x7f6f0c06ee40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6f24006fd0 tx=0x7f6f24008040 comp rx=0 tx=0).stop 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f280d0da0 msgr2=0x7f6f281513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f280d0da0 0x7f6f281513b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6f18000c00 tx=0x7f6f180046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 shutdown_connections 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f6f28160310 0x7f6f280c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f280d0da0 0x7f6f281513b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f6f0c06c980 0x7f6f0c06ee40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 --2- 172.21.15.12:0/227539689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f28070940 0x7f6f28150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 >> 172.21.15.12:0/227539689 conn(0x7f6f280b3f40 msgr2=0x7f6f2806cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:30.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 shutdown_connections 2024-04-15T15:23:30.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.675+0000 7f6f2fc41640 1 -- 172.21.15.12:0/227539689 wait complete. 2024-04-15T15:23:30.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:23:30.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.791+0000 7f13c93c8640 1 Processor -- start 2024-04-15T15:23:30.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.791+0000 7f13c93c8640 1 -- start start 2024-04-15T15:23:30.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c93c8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f13c40d0690 0x7f13c40d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c93c8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c40d0fb0 con 0x7f13c40b3160 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c93c8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c415cb50 con 0x7f13c40ae280 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c93c8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f13c415bda0 con 0x7f13c40d0690 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c2ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f13c40d0690 0x7f13c40d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c2ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f13c40d0690 0x7f13c40d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46280/0 (socket says 172.21.15.12:46280) 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c2ffd640 1 -- 172.21.15.12:0/3226769892 learned_addr learned my addr 172.21.15.12:0/3226769892 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c2ffd640 1 -- 172.21.15.12:0/3226769892 >> v1:172.21.15.55:6789/0 conn(0x7f13c40ae280 legacy=0x7f13c40ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c2ffd640 1 -- 172.21.15.12:0/3226769892 >> v1:172.21.15.12:6789/0 conn(0x7f13c40b3160 legacy=0x7f13c415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.792+0000 7f13c2ffd640 1 -- 172.21.15.12:0/3226769892 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c415bfd0 con 0x7f13c40d0690 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.793+0000 7f13c2ffd640 1 --2- 172.21.15.12:0/3226769892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40d0690 0x7f13c40d0a70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009870 tx=0x7f13b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533fe84b44592451 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.793+0000 7f13c27fc640 1 -- 172.21.15.12:0/3226769892 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4034070 con 0x7f13c40d0690 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.793+0000 7f13c27fc640 1 -- 172.21.15.12:0/3226769892 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b4030860 con 0x7f13c40d0690 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.793+0000 7f13c27fc640 1 -- 172.21.15.12:0/3226769892 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4030c30 con 0x7f13c40d0690 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.793+0000 7f13c93c8640 1 -- 172.21.15.12:0/3226769892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40d0690 msgr2=0x7f13c40d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.794+0000 7f13c93c8640 1 --2- 172.21.15.12:0/3226769892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40d0690 0x7f13c40d0a70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009870 tx=0x7f13b402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.794+0000 7f13c93c8640 1 -- 172.21.15.12:0/3226769892 shutdown_connections 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.794+0000 7f13c93c8640 1 --2- 172.21.15.12:0/3226769892 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40d0690 0x7f13c40d0a70 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.794+0000 7f13c93c8640 1 -- 172.21.15.12:0/3226769892 >> 172.21.15.12:0/3226769892 conn(0x7f13c406be00 msgr2=0x7f13c406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.794+0000 7f13c93c8640 1 -- 172.21.15.12:0/3226769892 shutdown_connections 2024-04-15T15:23:30.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.794+0000 7f13c93c8640 1 -- 172.21.15.12:0/3226769892 wait complete. 2024-04-15T15:23:30.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.795+0000 7f13c93c8640 1 Processor -- start 2024-04-15T15:23:30.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.795+0000 7f13c93c8640 1 -- start start 2024-04-15T15:23:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.795+0000 7f13c93c8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13c40ae280 0x7f13c4152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.795+0000 7f13c93c8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40b3160 0x7f13c40ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13c93c8640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f13c40d0690 0x7f13c415f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13c93c8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f13c415c870 con 0x7f13c40b3160 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13c93c8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f13c415c590 con 0x7f13c40ae280 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13c93c8640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f13c40b4fb0 con 0x7f13c40d0690 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40b3160 0x7f13c40ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40b3160 0x7f13c40ceb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46296/0 (socket says 172.21.15.12:46296) 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 -- 172.21.15.12:0/3874505929 learned_addr learned my addr 172.21.15.12:0/3874505929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 -- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f13c40d0690 msgr2=0x7f13c415f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f13c40d0690 0x7f13c415f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 -- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13c40ae280 msgr2=0x7f13c4152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13c40ae280 0x7f13c4152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 -- 172.21.15.12:0/3874505929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c415fd30 con 0x7f13c40b3160 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bbfff640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40b3160 0x7f13c40ceb80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00c3f0 tx=0x7f13ac00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bb7fe640 1 -- 172.21.15.12:0/3874505929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0052d0 con 0x7f13c40b3160 2024-04-15T15:23:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bb7fe640 1 -- 172.21.15.12:0/3874505929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ac005490 con 0x7f13c40b3160 2024-04-15T15:23:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13bb7fe640 1 -- 172.21.15.12:0/3874505929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac007260 con 0x7f13c40b3160 2024-04-15T15:23:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c40cddc0 con 0x7f13c40b3160 2024-04-15T15:23:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.796+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c40ce380 con 0x7f13c40b3160 2024-04-15T15:23:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.797+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c40689e0 con 0x7f13c40b3160 2024-04-15T15:23:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.798+0000 7f13bb7fe640 1 -- 172.21.15.12:0/3874505929 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f13ac007400 con 0x7f13c40b3160 2024-04-15T15:23:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.798+0000 7f13bb7fe640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f13a406c980 0x7f13a406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.798+0000 7f13bb7fe640 1 -- 172.21.15.12:0/3874505929 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f13ac08b3a0 con 0x7f13c40b3160 2024-04-15T15:23:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.799+0000 7f13c2ffd640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f13a406c980 0x7f13a406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:30.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.799+0000 7f13c2ffd640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f13a406c980 0x7f13a406ee40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f13b4036000 tx=0x7f13b4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:30.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:30.803+0000 7f13bb7fe640 1 -- 172.21.15.12:0/3874505929 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13ac0594e0 con 0x7f13c40b3160 2024-04-15T15:23:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.027+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13c406dc30 con 0x7f13c40b3160 2024-04-15T15:23:31.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.027+0000 7f13bb7fe640 1 -- 172.21.15.12:0/3874505929 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5180 (secure 0 0 0) 0x7f13ac058de0 con 0x7f13c40b3160 2024-04-15T15:23:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:31.029 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5102,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/852515692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":852515692},{"type":"v1","addr":"172.21.15.55:6837","nonce":852515692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:22:32.656644+0000","modified":"2024-04-15T15:23:19.042212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5200":{"gid":5200,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.55:6836/1202155349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1202155349},{"type":"v1","addr":"172.21.15.55:6836","nonce":1202155349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5200,"qdb_cluster":[5200]},"id":2}]} 2024-04-15T15:23:31.029 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-15T15:23:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f13a406c980 msgr2=0x7f13a406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f13a406c980 0x7f13a406ee40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f13b4036000 tx=0x7f13b4030000 comp rx=0 tx=0).stop 2024-04-15T15:23:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40b3160 msgr2=0x7f13c40ceb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40b3160 0x7f13c40ceb80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00c3f0 tx=0x7f13ac00c8b0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 shutdown_connections 2024-04-15T15:23:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f13c40d0690 0x7f13c415f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c40b3160 0x7f13c40ceb80 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f13a406c980 0x7f13a406ee40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 --2- 172.21.15.12:0/3874505929 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13c40ae280 0x7f13c4152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.029+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 >> 172.21.15.12:0/3874505929 conn(0x7f13c406be00 msgr2=0x7f13c406e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.030+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 shutdown_connections 2024-04-15T15:23:31.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.030+0000 7f13c93c8640 1 -- 172.21.15.12:0/3874505929 wait complete. 2024-04-15T15:23:31.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:23:31.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.148+0000 7f2884c0c640 1 Processor -- start 2024-04-15T15:23:31.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.148+0000 7f2884c0c640 1 -- start start 2024-04-15T15:23:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.149+0000 7f2884c0c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f288015e990 0x7f288006e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.149+0000 7f2884c0c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f288009a9d0 0x7f288009ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.149+0000 7f2884c0c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288006e9d0 con 0x7f28800b4ab0 2024-04-15T15:23:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.149+0000 7f2884c0c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f288015ed70 con 0x7f288009a9d0 2024-04-15T15:23:31.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.149+0000 7f2884c0c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f288006e000 con 0x7f288015e990 2024-04-15T15:23:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.149+0000 7f287ed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f288009a9d0 0x7f288009ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.149+0000 7f287ed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f288009a9d0 0x7f288009ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46304/0 (socket says 172.21.15.12:46304) 2024-04-15T15:23:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.149+0000 7f287ed9d640 1 -- 172.21.15.12:0/2022309614 learned_addr learned my addr 172.21.15.12:0/2022309614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.150+0000 7f287e59c640 1 --2- 172.21.15.12:0/2022309614 >> v2:172.21.15.55:3300/0 conn(0x7f288015e990 0x7f288006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.150+0000 7f287ed9d640 1 -- 172.21.15.12:0/2022309614 >> v2:172.21.15.55:3300/0 conn(0x7f288015e990 msgr2=0x7f288006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.150+0000 7f287ed9d640 1 --2- 172.21.15.12:0/2022309614 >> v2:172.21.15.55:3300/0 conn(0x7f288015e990 0x7f288006e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.150+0000 7f287ed9d640 1 -- 172.21.15.12:0/2022309614 >> v1:172.21.15.12:6789/0 conn(0x7f28800b4ab0 legacy=0x7f288009b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.150+0000 7f287ed9d640 1 -- 172.21.15.12:0/2022309614 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f288006d3f0 con 0x7f288009a9d0 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.150+0000 7f287ed9d640 1 --2- 172.21.15.12:0/2022309614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f288009ae00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2880073540 tx=0x7f287002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b7368a0fe1c14a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f287d59a640 1 -- 172.21.15.12:0/2022309614 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870034070 con 0x7f288009a9d0 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f287d59a640 1 -- 172.21.15.12:0/2022309614 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2870030860 con 0x7f288009a9d0 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f287d59a640 1 -- 172.21.15.12:0/2022309614 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870030c30 con 0x7f288009a9d0 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f2884c0c640 1 -- 172.21.15.12:0/2022309614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 msgr2=0x7f288009ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f2884c0c640 1 --2- 172.21.15.12:0/2022309614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f288009ae00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2880073540 tx=0x7f287002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f2884c0c640 1 -- 172.21.15.12:0/2022309614 shutdown_connections 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f2884c0c640 1 --2- 172.21.15.12:0/2022309614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f288009ae00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f2884c0c640 1 --2- 172.21.15.12:0/2022309614 >> v2:172.21.15.55:3300/0 conn(0x7f288015e990 0x7f288006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f2884c0c640 1 -- 172.21.15.12:0/2022309614 >> 172.21.15.12:0/2022309614 conn(0x7f28800b3eb0 msgr2=0x7f28800b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:31.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f2884c0c640 1 -- 172.21.15.12:0/2022309614 shutdown_connections 2024-04-15T15:23:31.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.151+0000 7f2884c0c640 1 -- 172.21.15.12:0/2022309614 wait complete. 2024-04-15T15:23:31.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f2884c0c640 1 Processor -- start 2024-04-15T15:23:31.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f2884c0c640 1 -- start start 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f2884c0c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f2880150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f2884c0c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28800b4ab0 0x7f28800c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f2884c0c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f288015e990 0x7f28800c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f2884c0c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f288006d9b0 con 0x7f288009a9d0 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f2884c0c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f288009b340 con 0x7f288015e990 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f2884c0c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f288006dc90 con 0x7f28800b4ab0 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f287e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f2880150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f287e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f2880150280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46312/0 (socket says 172.21.15.12:46312) 2024-04-15T15:23:31.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.153+0000 7f287e59c640 1 -- 172.21.15.12:0/1951053444 learned_addr learned my addr 172.21.15.12:0/1951053444 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f287ed9d640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f288015e990 0x7f28800c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f287e59c640 1 -- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28800b4ab0 msgr2=0x7f28800c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f287e59c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28800b4ab0 0x7f28800c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f287e59c640 1 -- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f288015e990 msgr2=0x7f28800c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f287e59c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f288015e990 0x7f28800c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f287e59c640 1 -- 172.21.15.12:0/1951053444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28800b6800 con 0x7f288009a9d0 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f287ed9d640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f288015e990 0x7f28800c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f287e59c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f2880150280 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2878007ef0 tx=0x7f287800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f2876ffd640 1 -- 172.21.15.12:0/1951053444 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2878014620 con 0x7f288009a9d0 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f2876ffd640 1 -- 172.21.15.12:0/1951053444 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2878019070 con 0x7f288009a9d0 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28800b6ae0 con 0x7f288009a9d0 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f2876ffd640 1 -- 172.21.15.12:0/1951053444 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2878014a00 con 0x7f288009a9d0 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.154+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28800b70a0 con 0x7f288009a9d0 2024-04-15T15:23:31.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.155+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28800689e0 con 0x7f288009a9d0 2024-04-15T15:23:31.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.155+0000 7f2876ffd640 1 -- 172.21.15.12:0/1951053444 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2878014ba0 con 0x7f288009a9d0 2024-04-15T15:23:31.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.156+0000 7f2876ffd640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f286006c980 0x7f286006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.156+0000 7f2876ffd640 1 -- 172.21.15.12:0/1951053444 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f287808c620 con 0x7f288009a9d0 2024-04-15T15:23:31.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.156+0000 7f287dd9b640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f286006c980 0x7f286006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.157+0000 7f287dd9b640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f286006c980 0x7f286006ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2868000f60 tx=0x7f28680079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.161+0000 7f2876ffd640 1 -- 172.21.15.12:0/1951053444 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f287805a8f0 con 0x7f288009a9d0 2024-04-15T15:23:31.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.347+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28800d0ab0 con 0x7f288009a9d0 2024-04-15T15:23:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.348+0000 7f2876ffd640 1 -- 172.21.15.12:0/1951053444 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17084 (secure 0 0 0) 0x7f287805a1f0 con 0x7f288009a9d0 2024-04-15T15:23:31.350 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:31.350 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":42,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:23:15.001862+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:22:31.098808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:22:31.597572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6837/3388147599":"2024-04-16T15:23:15.001839+0000","172.21.15.55:6835/3388147599":"2024-04-16T15:23:15.001839+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:23:31.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f286006c980 msgr2=0x7f286006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f286006c980 0x7f286006ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2868000f60 tx=0x7f28680079d0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 msgr2=0x7f2880150280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f2880150280 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2878007ef0 tx=0x7f287800e4b0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 shutdown_connections 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f286006c980 0x7f286006ee40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f288015e990 0x7f28800c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f28800b4ab0 0x7f28800c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 --2- 172.21.15.12:0/1951053444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f288009a9d0 0x7f2880150280 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 >> 172.21.15.12:0/1951053444 conn(0x7f28800b3eb0 msgr2=0x7f2880070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 shutdown_connections 2024-04-15T15:23:31.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.350+0000 7f2884c0c640 1 -- 172.21.15.12:0/1951053444 wait complete. 2024-04-15T15:23:31.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:23:31.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.467+0000 7f3760d66640 1 Processor -- start 2024-04-15T15:23:31.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.467+0000 7f3760d66640 1 -- start start 2024-04-15T15:23:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f3760d66640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f375c15e790 0x7f375c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f3760d66640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f3760d66640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f375c0d2480 con 0x7f375c0d0f00 2024-04-15T15:23:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f3760d66640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f375c0d2db0 con 0x7f375c15e790 2024-04-15T15:23:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f3760d66640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f375c0d2760 con 0x7f375c0cfd20 2024-04-15T15:23:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f375a59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f375c15e790 0x7f375c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f375a59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f375c15e790 0x7f375c0d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46322/0 (socket says 172.21.15.12:46322) 2024-04-15T15:23:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f375ad9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f375c0d0f00 0x7f375c0b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47142/0 (socket says 172.21.15.12:47142) 2024-04-15T15:23:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f375a59c640 1 -- 172.21.15.12:0/4017604282 learned_addr learned my addr 172.21.15.12:0/4017604282 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.468+0000 7f3759d9b640 1 --2- 172.21.15.12:0/4017604282 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.469+0000 7f375a59c640 1 -- 172.21.15.12:0/4017604282 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 msgr2=0x7f375c0d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.469+0000 7f375a59c640 1 --2- 172.21.15.12:0/4017604282 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c0d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.469+0000 7f375a59c640 1 -- 172.21.15.12:0/4017604282 >> v1:172.21.15.12:6789/0 conn(0x7f375c0d0f00 legacy=0x7f375c0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.469+0000 7f375a59c640 1 -- 172.21.15.12:0/4017604282 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c0d2a40 con 0x7f375c15e790 2024-04-15T15:23:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.469+0000 7f375a59c640 1 --2- 172.21.15.12:0/4017604282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c15e790 0x7f375c0d3240 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3754009870 tx=0x7f375402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be886086dcf7303 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f375959a640 1 -- 172.21.15.12:0/4017604282 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754034070 con 0x7f375c15e790 2024-04-15T15:23:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f375959a640 1 -- 172.21.15.12:0/4017604282 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3754030840 con 0x7f375c15e790 2024-04-15T15:23:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f375959a640 1 -- 172.21.15.12:0/4017604282 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754030c10 con 0x7f375c15e790 2024-04-15T15:23:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f3760d66640 1 -- 172.21.15.12:0/4017604282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c15e790 msgr2=0x7f375c0d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f3760d66640 1 --2- 172.21.15.12:0/4017604282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c15e790 0x7f375c0d3240 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3754009870 tx=0x7f375402b980 comp rx=0 tx=0).stop 2024-04-15T15:23:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f3760d66640 1 -- 172.21.15.12:0/4017604282 shutdown_connections 2024-04-15T15:23:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f3760d66640 1 --2- 172.21.15.12:0/4017604282 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f3760d66640 1 --2- 172.21.15.12:0/4017604282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c15e790 0x7f375c0d3240 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f3760d66640 1 -- 172.21.15.12:0/4017604282 >> 172.21.15.12:0/4017604282 conn(0x7f375c08bf60 msgr2=0x7f375c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f3760d66640 1 -- 172.21.15.12:0/4017604282 shutdown_connections 2024-04-15T15:23:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.470+0000 7f3760d66640 1 -- 172.21.15.12:0/4017604282 wait complete. 2024-04-15T15:23:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.471+0000 7f3760d66640 1 Processor -- start 2024-04-15T15:23:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3760d66640 1 -- start start 2024-04-15T15:23:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3760d66640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c14b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3760d66640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c0d0f00 0x7f375c0c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3760d66640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f375c15e790 0x7f375c0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3760d66640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f375c15ecf0 con 0x7f375c0d0f00 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3760d66640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f375c15eb70 con 0x7f375c15e790 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3760d66640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f375c0d13b0 con 0x7f375c0cfd20 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3759d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c0d0f00 0x7f375c0c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3759d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c0d0f00 0x7f375c0c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46330/0 (socket says 172.21.15.12:46330) 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.472+0000 7f3759d9b640 1 -- 172.21.15.12:0/1853428677 learned_addr learned my addr 172.21.15.12:0/1853428677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f375ad9d640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f375c15e790 0x7f375c0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f375a59c640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c14b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3759d9b640 1 -- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 msgr2=0x7f375c14b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3759d9b640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c14b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3759d9b640 1 -- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f375c15e790 msgr2=0x7f375c0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3759d9b640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f375c15e790 0x7f375c0ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3759d9b640 1 -- 172.21.15.12:0/1853428677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c15a360 con 0x7f375c0d0f00 2024-04-15T15:23:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f375ad9d640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f375c15e790 0x7f375c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f375a59c640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c14b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3759d9b640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c0d0f00 0x7f375c0c9000 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f374400e9b0 tx=0x7f374400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3752ffd640 1 -- 172.21.15.12:0/1853428677 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744017070 con 0x7f375c0d0f00 2024-04-15T15:23:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3752ffd640 1 -- 172.21.15.12:0/1853428677 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374400c9a0 con 0x7f375c0d0f00 2024-04-15T15:23:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375c15a640 con 0x7f375c0d0f00 2024-04-15T15:23:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3752ffd640 1 -- 172.21.15.12:0/1853428677 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374400cd90 con 0x7f375c0d0f00 2024-04-15T15:23:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.473+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375c15aee0 con 0x7f375c0d0f00 2024-04-15T15:23:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.474+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f375c0689e0 con 0x7f375c0d0f00 2024-04-15T15:23:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.475+0000 7f3752ffd640 1 -- 172.21.15.12:0/1853428677 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f374401a070 con 0x7f375c0d0f00 2024-04-15T15:23:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.475+0000 7f3752ffd640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f373c06c9b0 0x7f373c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.475+0000 7f3752ffd640 1 -- 172.21.15.12:0/1853428677 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f374408bd80 con 0x7f375c0d0f00 2024-04-15T15:23:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.475+0000 7f375a59c640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f373c06c9b0 0x7f373c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.476+0000 7f375a59c640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f373c06c9b0 0x7f373c06ee70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3754036000 tx=0x7f375403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.479+0000 7f3752ffd640 1 -- 172.21.15.12:0/1853428677 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3744059fc0 con 0x7f375c0d0f00 2024-04-15T15:23:31.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.708+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f375c0d1ce0 con 0x7f375c0d0f00 2024-04-15T15:23:31.710 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-04-15T15:23:31.709+0000 7fbdbcac3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:23:31.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.710+0000 7f3752ffd640 1 -- 172.21.15.12:0/1853428677 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7f37440598c0 con 0x7f375c0d0f00 2024-04-15T15:23:31.711 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:23:31.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f373c06c9b0 msgr2=0x7f373c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f373c06c9b0 0x7f373c06ee70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3754036000 tx=0x7f375403b040 comp rx=0 tx=0).stop 2024-04-15T15:23:31.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c0d0f00 msgr2=0x7f375c0c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c0d0f00 0x7f375c0c9000 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f374400e9b0 tx=0x7f374400ee70 comp rx=0 tx=0).stop 2024-04-15T15:23:31.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 shutdown_connections 2024-04-15T15:23:31.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f373c06c9b0 0x7f373c06ee70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f375c15e790 0x7f375c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375c0d0f00 0x7f375c0c9000 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 --2- 172.21.15.12:0/1853428677 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f375c0cfd20 0x7f375c14b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.712+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 >> 172.21.15.12:0/1853428677 conn(0x7f375c08bf60 msgr2=0x7f375c0caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:31.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.713+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 shutdown_connections 2024-04-15T15:23:31.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.713+0000 7f3760d66640 1 -- 172.21.15.12:0/1853428677 wait complete. 2024-04-15T15:23:31.725 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-15T15:23:31.726 INFO:tasks.ceph.mds.b.smithi055.stderr: -803> 2024-04-15T15:23:23.080+0000 7f1eaa5e8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:23:31.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:23:31.754 INFO:tasks.ceph.mds.b.smithi055.stdout:starting mds.b at 2024-04-15T15:23:31.772 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:23:31.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.832+0000 7f4440731640 1 Processor -- start 2024-04-15T15:23:31.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.832+0000 7f4440731640 1 -- start start 2024-04-15T15:23:31.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.833+0000 7f4440731640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f443815b520 0x7f44380ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.833+0000 7f4440731640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f44380cee70 0x7f44380ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.833+0000 7f4440731640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44380cbee0 con 0x7f44380cf650 2024-04-15T15:23:31.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.833+0000 7f4440731640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f44380cc1c0 con 0x7f443815b520 2024-04-15T15:23:31.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.833+0000 7f4440731640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f44380cc4a0 con 0x7f44380cee70 2024-04-15T15:23:31.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.833+0000 7f443e4cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f443815b520 0x7f44380ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.833+0000 7f443e4cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f443815b520 0x7f44380ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46342/0 (socket says 172.21.15.12:46342) 2024-04-15T15:23:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.833+0000 7f443e4cd640 1 -- 172.21.15.12:0/2786715663 learned_addr learned my addr 172.21.15.12:0/2786715663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.834+0000 7f443e4cd640 1 -- 172.21.15.12:0/2786715663 >> v2:172.21.15.55:3300/0 conn(0x7f44380cee70 msgr2=0x7f44380ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.834+0000 7f443e4cd640 1 --2- 172.21.15.12:0/2786715663 >> v2:172.21.15.55:3300/0 conn(0x7f44380cee70 0x7f44380ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.834+0000 7f443e4cd640 1 -- 172.21.15.12:0/2786715663 >> v1:172.21.15.55:6789/0 conn(0x7f44380cf650 legacy=0x7f443806eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.834+0000 7f443e4cd640 1 -- 172.21.15.12:0/2786715663 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44380cc780 con 0x7f443815b520 2024-04-15T15:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.834+0000 7f443e4cd640 1 --2- 172.21.15.12:0/2786715663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f443815b520 0x7f44380ccca0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4438072320 tx=0x7f443002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d25481380aa3647 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f443d4cb640 1 -- 172.21.15.12:0/2786715663 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443003b070 con 0x7f443815b520 2024-04-15T15:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f443d4cb640 1 -- 172.21.15.12:0/2786715663 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4430036440 con 0x7f443815b520 2024-04-15T15:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f443d4cb640 1 -- 172.21.15.12:0/2786715663 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430036810 con 0x7f443815b520 2024-04-15T15:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f4440731640 1 -- 172.21.15.12:0/2786715663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f443815b520 msgr2=0x7f44380ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f4440731640 1 --2- 172.21.15.12:0/2786715663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f443815b520 0x7f44380ccca0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4438072320 tx=0x7f443002e0e0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f4440731640 1 -- 172.21.15.12:0/2786715663 shutdown_connections 2024-04-15T15:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f4440731640 1 --2- 172.21.15.12:0/2786715663 >> v2:172.21.15.55:3300/0 conn(0x7f44380cee70 0x7f44380ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f4440731640 1 --2- 172.21.15.12:0/2786715663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f443815b520 0x7f44380ccca0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f4440731640 1 -- 172.21.15.12:0/2786715663 >> 172.21.15.12:0/2786715663 conn(0x7f44380b2e70 msgr2=0x7f44380b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f4440731640 1 -- 172.21.15.12:0/2786715663 shutdown_connections 2024-04-15T15:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.835+0000 7f4440731640 1 -- 172.21.15.12:0/2786715663 wait complete. 2024-04-15T15:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.836+0000 7f4440731640 1 Processor -- start 2024-04-15T15:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.836+0000 7f4440731640 1 -- start start 2024-04-15T15:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f4440731640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44380cee70 0x7f44381f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f4440731640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f44380cf650 0x7f443814e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f4440731640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f443815b520 0x7f44380cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f4440731640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f443815b900 con 0x7f44380cee70 2024-04-15T15:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f4440731640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f44380cf3a0 con 0x7f443815b520 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f4440731640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f44381f42d0 con 0x7f44380cf650 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f443e4cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44380cee70 0x7f44381f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f443e4cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44380cee70 0x7f44381f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46356/0 (socket says 172.21.15.12:46356) 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f443e4cd640 1 -- 172.21.15.12:0/3975008161 learned_addr learned my addr 172.21.15.12:0/3975008161 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f443e4cd640 1 -- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f44380cf650 msgr2=0x7f443814e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.837+0000 7f443e4cd640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f44380cf650 0x7f443814e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f443e4cd640 1 -- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f443815b520 msgr2=0x7f44380cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f443ecce640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f443815b520 0x7f44380cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f443e4cd640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f443815b520 0x7f44380cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f443e4cd640 1 -- 172.21.15.12:0/3975008161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443814e5d0 con 0x7f44380cee70 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f443e4cd640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44380cee70 0x7f44381f4760 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f443002e120 tx=0x7f44300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f4436ffd640 1 -- 172.21.15.12:0/3975008161 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430045070 con 0x7f44380cee70 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f4436ffd640 1 -- 172.21.15.12:0/3975008161 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4430040890 con 0x7f44380cee70 2024-04-15T15:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44380c8370 con 0x7f44380cee70 2024-04-15T15:23:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f4436ffd640 1 -- 172.21.15.12:0/3975008161 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443003b040 con 0x7f44380cee70 2024-04-15T15:23:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.838+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44380c8930 con 0x7f44380cee70 2024-04-15T15:23:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.839+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44380690f0 con 0x7f44380cee70 2024-04-15T15:23:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.839+0000 7f4436ffd640 1 -- 172.21.15.12:0/3975008161 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4430048070 con 0x7f44380cee70 2024-04-15T15:23:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.840+0000 7f4436ffd640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f442406c9b0 0x7f442406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.840+0000 7f4436ffd640 1 -- 172.21.15.12:0/3975008161 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6302+0+0 (secure 0 0 0) 0x7f44300b0c20 con 0x7f44380cee70 2024-04-15T15:23:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.840+0000 7f443dccc640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f442406c9b0 0x7f442406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.840+0000 7f443dccc640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f442406c9b0 0x7f442406ee70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f44381f3be0 tx=0x7f442c006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:31.844+0000 7f4436ffd640 1 -- 172.21.15.12:0/3975008161 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f443007ef30 con 0x7f44380cee70 2024-04-15T15:23:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.073+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f44380c8b70 con 0x7f44380cee70 2024-04-15T15:23:32.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.711+0000 7f4436ffd640 1 -- 172.21.15.12:0/3975008161 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f443007e830 con 0x7f44380cee70 2024-04-15T15:23:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f442406c9b0 msgr2=0x7f442406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f442406c9b0 0x7f442406ee70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f44381f3be0 tx=0x7f442c006c90 comp rx=0 tx=0).stop 2024-04-15T15:23:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44380cee70 msgr2=0x7f44381f4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44380cee70 0x7f44381f4760 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f443002e120 tx=0x7f44300046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 shutdown_connections 2024-04-15T15:23:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f442406c9b0 0x7f442406ee70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f443815b520 0x7f44380cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f44380cf650 0x7f443814e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 --2- 172.21.15.12:0/3975008161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44380cee70 0x7f44381f4760 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 >> 172.21.15.12:0/3975008161 conn(0x7f44380b2e70 msgr2=0x7f443815a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:32.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 shutdown_connections 2024-04-15T15:23:32.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.714+0000 7f4440731640 1 -- 172.21.15.12:0/3975008161 wait complete. 2024-04-15T15:23:32.729 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-15T15:23:32.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.883+0000 7f3f249bd640 1 Processor -- start 2024-04-15T15:23:32.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.883+0000 7f3f249bd640 1 -- start start 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f249bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f200cd290 0x7f3f200c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f249bd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f200cdea0 con 0x7f3f2015acb0 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f249bd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f200ce180 con 0x7f3f200b4b50 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f249bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3f200ce460 con 0x7f3f200cd290 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f15d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f200cd290 0x7f3f200c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f15d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f200cd290 0x7f3f200c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53688/0 (socket says 172.21.15.12:53688) 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f15d9b640 1 -- 172.21.15.12:0/2965822261 learned_addr learned my addr 172.21.15.12:0/2965822261 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f15d9b640 1 -- 172.21.15.12:0/2965822261 >> v1:172.21.15.55:6789/0 conn(0x7f3f200b4b50 legacy=0x7f3f200c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f15d9b640 1 -- 172.21.15.12:0/2965822261 >> v1:172.21.15.12:6789/0 conn(0x7f3f2015acb0 legacy=0x7f3f200ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:32.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.884+0000 7f3f15d9b640 1 -- 172.21.15.12:0/2965822261 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f200b3c10 con 0x7f3f200cd290 2024-04-15T15:23:32.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.885+0000 7f3f15d9b640 1 --2- 172.21.15.12:0/2965822261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200cd290 0x7f3f200c50d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800bb00 tx=0x7f3f0802dfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb52af94d5c3c397 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:32.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.885+0000 7f3f1dd9b640 1 -- 172.21.15.12:0/2965822261 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f08034070 con 0x7f3f200cd290 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.885+0000 7f3f1dd9b640 1 -- 172.21.15.12:0/2965822261 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f0802ed90 con 0x7f3f200cd290 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.885+0000 7f3f1dd9b640 1 -- 172.21.15.12:0/2965822261 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f08038620 con 0x7f3f200cd290 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.885+0000 7f3f249bd640 1 -- 172.21.15.12:0/2965822261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200cd290 msgr2=0x7f3f200c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.885+0000 7f3f249bd640 1 --2- 172.21.15.12:0/2965822261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200cd290 0x7f3f200c50d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800bb00 tx=0x7f3f0802dfe0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.886+0000 7f3f249bd640 1 -- 172.21.15.12:0/2965822261 shutdown_connections 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.886+0000 7f3f249bd640 1 --2- 172.21.15.12:0/2965822261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200cd290 0x7f3f200c50d0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.886+0000 7f3f249bd640 1 -- 172.21.15.12:0/2965822261 >> 172.21.15.12:0/2965822261 conn(0x7f3f2006bbc0 msgr2=0x7f3f2006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.886+0000 7f3f249bd640 1 -- 172.21.15.12:0/2965822261 shutdown_connections 2024-04-15T15:23:32.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.886+0000 7f3f249bd640 1 -- 172.21.15.12:0/2965822261 wait complete. 2024-04-15T15:23:32.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.887+0000 7f3f249bd640 1 Processor -- start 2024-04-15T15:23:32.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.887+0000 7f3f249bd640 1 -- start start 2024-04-15T15:23:32.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f249bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200b4b50 0x7f3f201effd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:32.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f249bd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f200cd290 0x7f3f20152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:32.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f249bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3f2015acb0 0x7f3f201582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f249bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2015b090 con 0x7f3f200b4b50 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f249bd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f200cfed0 con 0x7f3f200cd290 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f249bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f3f2015b210 con 0x7f3f2015acb0 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200b4b50 0x7f3f201effd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200b4b50 0x7f3f201effd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53690/0 (socket says 172.21.15.12:53690) 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 -- 172.21.15.12:0/1256055207 learned_addr learned my addr 172.21.15.12:0/1256055207 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f15d9b640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f200cd290 0x7f3f20152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1ed9d640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3f2015acb0 0x7f3f201582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 -- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3f2015acb0 msgr2=0x7f3f201582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3f2015acb0 0x7f3f201582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 -- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f200cd290 msgr2=0x7f3f20152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f200cd290 0x7f3f20152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 -- 172.21.15.12:0/1256055207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f20153f00 con 0x7f3f200b4b50 2024-04-15T15:23:32.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.888+0000 7f3f1e59c640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200b4b50 0x7f3f201effd0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000c430 tx=0x7f3f1000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:32.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.889+0000 7f3f177fe640 1 -- 172.21.15.12:0/1256055207 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10017070 con 0x7f3f200b4b50 2024-04-15T15:23:32.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.889+0000 7f3f177fe640 1 -- 172.21.15.12:0/1256055207 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f100052d0 con 0x7f3f200b4b50 2024-04-15T15:23:32.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.889+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f20153940 con 0x7f3f200b4b50 2024-04-15T15:23:32.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.889+0000 7f3f177fe640 1 -- 172.21.15.12:0/1256055207 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10003830 con 0x7f3f200b4b50 2024-04-15T15:23:32.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.889+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f20155fb0 con 0x7f3f200b4b50 2024-04-15T15:23:32.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.890+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f200689e0 con 0x7f3f200b4b50 2024-04-15T15:23:32.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.890+0000 7f3f177fe640 1 -- 172.21.15.12:0/1256055207 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3f1001a070 con 0x7f3f200b4b50 2024-04-15T15:23:32.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.891+0000 7f3f177fe640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3f0006c980 0x7f3f0006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:32.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.891+0000 7f3f177fe640 1 -- 172.21.15.12:0/1256055207 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6302+0+0 (secure 0 0 0) 0x7f3f1008b090 con 0x7f3f200b4b50 2024-04-15T15:23:32.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.891+0000 7f3f15d9b640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3f0006c980 0x7f3f0006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:32.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.891+0000 7f3f15d9b640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3f0006c980 0x7f3f0006ee40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3f08039000 tx=0x7f3f08034000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:32.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:32.895+0000 7f3f177fe640 1 -- 172.21.15.12:0/1256055207 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f10058c30 con 0x7f3f200b4b50 2024-04-15T15:23:33.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.079+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3f200cc450 con 0x7f3f200b4b50 2024-04-15T15:23:33.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.713+0000 7f3f177fe640 1 -- 172.21.15.12:0/1256055207 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) ==== 181+0+0 (secure 0 0 0) 0x7f3f1005cf30 con 0x7f3f200b4b50 2024-04-15T15:23:33.714 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-04-15T15:23:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3f0006c980 msgr2=0x7f3f0006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3f0006c980 0x7f3f0006ee40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3f08039000 tx=0x7f3f08034000 comp rx=0 tx=0).stop 2024-04-15T15:23:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200b4b50 msgr2=0x7f3f201effd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200b4b50 0x7f3f201effd0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000c430 tx=0x7f3f1000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 shutdown_connections 2024-04-15T15:23:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f3f2015acb0 0x7f3f201582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f3f0006c980 0x7f3f0006ee40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f200cd290 0x7f3f20152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 --2- 172.21.15.12:0/1256055207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f200b4b50 0x7f3f201effd0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 >> 172.21.15.12:0/1256055207 conn(0x7f3f2006bbc0 msgr2=0x7f3f200ad140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:33.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 shutdown_connections 2024-04-15T15:23:33.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.716+0000 7f3f249bd640 1 -- 172.21.15.12:0/1256055207 wait complete. 2024-04-15T15:23:33.732 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-15T15:23:33.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.845+0000 7f4f1c7bd640 1 Processor -- start 2024-04-15T15:23:33.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.845+0000 7f4f1c7bd640 1 -- start start 2024-04-15T15:23:33.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1c7bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f1415e7a0 0x7f4f140d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:33.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1c7bd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4f140d1870 0x7f4f14162a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:33.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1c7bd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f140d0530 con 0x7f4f140d13f0 2024-04-15T15:23:33.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1c7bd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4f140cf490 con 0x7f4f140d1870 2024-04-15T15:23:33.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1c7bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f4f140cf770 con 0x7f4f1415e7a0 2024-04-15T15:23:33.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1a559640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f1415e7a0 0x7f4f140d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:33.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1a559640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f1415e7a0 0x7f4f140d0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:43808/0 (socket says 172.21.15.12:43808) 2024-04-15T15:23:33.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1a559640 1 -- 172.21.15.12:0/2349692915 learned_addr learned my addr 172.21.15.12:0/2349692915 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:33.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f19d58640 1 --1- 172.21.15.12:0/2349692915 >> v1:172.21.15.55:6789/0 conn(0x7f4f140d13f0 0x7f4f14162370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:43460/0 (socket says 172.21.15.12:43460) 2024-04-15T15:23:33.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.846+0000 7f4f1ad5a640 1 --2- 172.21.15.12:0/2349692915 >> v2:172.21.15.55:3300/0 conn(0x7f4f140d1870 0x7f4f14162a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:33.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.847+0000 7f4f1a559640 1 -- 172.21.15.12:0/2349692915 >> v2:172.21.15.55:3300/0 conn(0x7f4f140d1870 msgr2=0x7f4f14162a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:33.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.847+0000 7f4f1a559640 1 --2- 172.21.15.12:0/2349692915 >> v2:172.21.15.55:3300/0 conn(0x7f4f140d1870 0x7f4f14162a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.847+0000 7f4f1a559640 1 -- 172.21.15.12:0/2349692915 >> v1:172.21.15.55:6789/0 conn(0x7f4f140d13f0 legacy=0x7f4f14162370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:33.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.847+0000 7f4f19557640 1 -- 172.21.15.12:0/2349692915 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893570482 0 0) 0x7f4f140d0530 con 0x7f4f140d13f0 2024-04-15T15:23:33.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.847+0000 7f4f1a559640 1 -- 172.21.15.12:0/2349692915 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f140cfa50 con 0x7f4f1415e7a0 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1a559640 1 --2- 172.21.15.12:0/2349692915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f1415e7a0 0x7f4f140d0eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f10009870 tx=0x7f4f1002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=785b9240a6514d7e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f19557640 1 -- 172.21.15.12:0/2349692915 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f10035020 con 0x7f4f1415e7a0 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f19557640 1 -- 172.21.15.12:0/2349692915 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f10031640 con 0x7f4f1415e7a0 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f19557640 1 -- 172.21.15.12:0/2349692915 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1002d040 con 0x7f4f1415e7a0 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/2349692915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f1415e7a0 msgr2=0x7f4f140d0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/2349692915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f1415e7a0 0x7f4f140d0eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f10009870 tx=0x7f4f1002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/2349692915 shutdown_connections 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/2349692915 >> v2:172.21.15.55:3300/0 conn(0x7f4f140d1870 0x7f4f14162a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/2349692915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f1415e7a0 0x7f4f140d0eb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/2349692915 >> 172.21.15.12:0/2349692915 conn(0x7f4f1408bf60 msgr2=0x7f4f140d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:33.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/2349692915 shutdown_connections 2024-04-15T15:23:33.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.848+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/2349692915 wait complete. 2024-04-15T15:23:33.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f1c7bd640 1 Processor -- start 2024-04-15T15:23:33.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f1c7bd640 1 -- start start 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f1c7bd640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f140d13f0 0x7f4f1415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f1c7bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f140d1870 0x7f4f140b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f1c7bd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f1415e7a0 0x7f4f14161970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f1c7bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1406db20 con 0x7f4f140d1870 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f1c7bd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4f140cf180 con 0x7f4f1415e7a0 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f1c7bd640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1415eb80 con 0x7f4f140d13f0 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f19d58640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f140d1870 0x7f4f140b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f19d58640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f140d1870 0x7f4f140b7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53696/0 (socket says 172.21.15.12:53696) 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.850+0000 7f4f19d58640 1 -- 172.21.15.12:0/3884206437 learned_addr learned my addr 172.21.15.12:0/3884206437 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f1a559640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f140d13f0 0x7f4f1415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f1ad5a640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f1415e7a0 0x7f4f14161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f19d58640 1 -- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f140d13f0 msgr2=0x7f4f1415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f19d58640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f140d13f0 0x7f4f1415acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f19d58640 1 -- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f1415e7a0 msgr2=0x7f4f14161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f19d58640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f1415e7a0 0x7f4f14161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f19d58640 1 -- 172.21.15.12:0/3884206437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f140b98f0 con 0x7f4f140d1870 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f19d58640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f140d1870 0x7f4f140b7140 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400c460 tx=0x7f4f0400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f0affd640 1 -- 172.21.15.12:0/3884206437 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f04018070 con 0x7f4f140d1870 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f0affd640 1 -- 172.21.15.12:0/3884206437 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f04007500 con 0x7f4f140d1870 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f140b9bd0 con 0x7f4f140d1870 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f0affd640 1 -- 172.21.15.12:0/3884206437 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f04004a20 con 0x7f4f140d1870 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.851+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f14161170 con 0x7f4f140d1870 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.852+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f140689e0 con 0x7f4f140d1870 2024-04-15T15:23:33.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.853+0000 7f4f0affd640 1 -- 172.21.15.12:0/3884206437 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4f0401b070 con 0x7f4f140d1870 2024-04-15T15:23:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.853+0000 7f4f0affd640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4ef806c980 0x7f4ef806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.853+0000 7f4f0affd640 1 -- 172.21.15.12:0/3884206437 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5864+0+0 (secure 0 0 0) 0x7f4f0408af20 con 0x7f4f140d1870 2024-04-15T15:23:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.853+0000 7f4f1a559640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4ef806c980 0x7f4ef806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.854+0000 7f4f1a559640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4ef806c980 0x7f4ef806ee40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4f10036000 tx=0x7f4f10008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:33.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:33.858+0000 7f4f0affd640 1 -- 172.21.15.12:0/3884206437 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f040592c0 con 0x7f4f140d1870 2024-04-15T15:23:34.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.040+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4f140d1d60 con 0x7f4f140d1870 2024-04-15T15:23:34.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.714+0000 7f4f0affd640 1 -- 172.21.15.12:0/3884206437 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) ==== 169+0+0 (secure 0 0 0) 0x7f4f04058bc0 con 0x7f4f140d1870 2024-04-15T15:23:34.716 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-04-15T15:23:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4ef806c980 msgr2=0x7f4ef806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:34.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4ef806c980 0x7f4ef806ee40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4f10036000 tx=0x7f4f10008030 comp rx=0 tx=0).stop 2024-04-15T15:23:34.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f140d1870 msgr2=0x7f4f140b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:34.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f140d1870 0x7f4f140b7140 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400c460 tx=0x7f4f0400c920 comp rx=0 tx=0).stop 2024-04-15T15:23:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 shutdown_connections 2024-04-15T15:23:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4ef806c980 0x7f4ef806ee40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f1415e7a0 0x7f4f14161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f140d1870 0x7f4f140b7140 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 --2- 172.21.15.12:0/3884206437 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f140d13f0 0x7f4f1415acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.718+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 >> 172.21.15.12:0/3884206437 conn(0x7f4f1408bf60 msgr2=0x7f4f140b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.719+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 shutdown_connections 2024-04-15T15:23:34.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.719+0000 7f4f1c7bd640 1 -- 172.21.15.12:0/3884206437 wait complete. 2024-04-15T15:23:34.735 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-04-15T15:23:34.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.846+0000 7f4f5ea63640 1 Processor -- start 2024-04-15T15:23:34.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.848+0000 7f4f5ea63640 1 -- start start 2024-04-15T15:23:34.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.849+0000 7f4f5ea63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f58162b10 0x7f4f5806f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.849+0000 7f4f5ea63640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4f580d1170 0x7f4f58162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.849+0000 7f4f5ea63640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f5806e5f0 con 0x7f4f5806e0a0 2024-04-15T15:23:34.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.849+0000 7f4f5ea63640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4f58072ee0 con 0x7f4f580d1170 2024-04-15T15:23:34.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.849+0000 7f4f5ea63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f4f58072120 con 0x7f4f58162b10 2024-04-15T15:23:34.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.851+0000 7f4f577fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f4f5806e0a0 0x7f4f580d0a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:43462/0 (socket says 172.21.15.12:43462) 2024-04-15T15:23:34.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.851+0000 7f4f577fe640 1 -- 172.21.15.12:0/4130736172 learned_addr learned my addr 172.21.15.12:0/4130736172 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:34.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.851+0000 7f4f57fff640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f58162b10 0x7f4f5806f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:34.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.852+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2873755381 0 0) 0x7f4f5806e5f0 con 0x7f4f5806e0a0 2024-04-15T15:23:34.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.852+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f40003520 con 0x7f4f5806e0a0 2024-04-15T15:23:34.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.852+0000 7f4f5d000640 1 --2- 172.21.15.12:0/4130736172 >> v2:172.21.15.55:3300/0 conn(0x7f4f580d1170 0x7f4f58162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:34.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.853+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 <== mon.1 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1794619604 0 0) 0x7f4f40003520 con 0x7f4f5806e0a0 2024-04-15T15:23:34.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.853+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f58162b10 msgr2=0x7f4f5806f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:34.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f58162b10 0x7f4f5806f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 >> v2:172.21.15.55:3300/0 conn(0x7f4f580d1170 msgr2=0x7f4f58162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:34.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 --2- 172.21.15.12:0/4130736172 >> v2:172.21.15.55:3300/0 conn(0x7f4f580d1170 0x7f4f58162ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f58072400 con 0x7f4f5806e0a0 2024-04-15T15:23:34.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 <== mon.1 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f4f4c001670 con 0x7f4f5806e0a0 2024-04-15T15:23:34.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 >> v1:172.21.15.55:6789/0 conn(0x7f4f5806e0a0 legacy=0x7f4f580d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:34.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f58162b10 0x7f4f400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f40003f80 0x7f4f40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f40004d40 0x7f4f400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4f58072ee0 con 0x7f4f40003f80 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4f58072120 con 0x7f4f40004d40 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f4f4000ba40 con 0x7f4f58162b10 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.854+0000 7f4f5d000640 1 --2- 172.21.15.12:0/4130736172 >> v2:172.21.15.55:3300/0 conn(0x7f4f580d1170 0x7f4f58162ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.855+0000 7f4f577fe640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f40003f80 0x7f4f40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.855+0000 7f4f5d000640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f40004d40 0x7f4f400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.855+0000 7f4f577fe640 1 -- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f58162b10 msgr2=0x7f4f400039b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.855+0000 7f4f577fe640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f58162b10 0x7f4f400039b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.855+0000 7f4f577fe640 1 -- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f40004d40 msgr2=0x7f4f400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.855+0000 7f4f577fe640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f40004d40 0x7f4f400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.855+0000 7f4f577fe640 1 -- 172.21.15.12:0/4130736172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f40003520 con 0x7f4f40003f80 2024-04-15T15:23:34.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.855+0000 7f4f5d000640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f40004d40 0x7f4f400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.857+0000 7f4f577fe640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f40003f80 0x7f4f40004800 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c007c20 tx=0x7f4f40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3338c602dbbd7db5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.857+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c007c60 con 0x7f4f40003f80 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.857+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f4c041070 con 0x7f4f40003f80 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.857+0000 7f4f56ffd640 1 -- 172.21.15.12:0/4130736172 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c005d70 con 0x7f4f40003f80 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.858+0000 7f4f5ea63640 1 -- 172.21.15.12:0/4130736172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f40003f80 msgr2=0x7f4f40004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.858+0000 7f4f5ea63640 1 --2- 172.21.15.12:0/4130736172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f40003f80 0x7f4f40004800 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c007c20 tx=0x7f4f40005830 comp rx=0 tx=0).stop 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.858+0000 7f4f5d000640 1 -- 172.21.15.12:0/4130736172 reap_dead start 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.858+0000 7f4f5ea63640 1 -- 172.21.15.12:0/4130736172 shutdown_connections 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.858+0000 7f4f5ea63640 1 -- 172.21.15.12:0/4130736172 >> 172.21.15.12:0/4130736172 conn(0x7f4f5806be00 msgr2=0x7f4f5806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.858+0000 7f4f5ea63640 1 -- 172.21.15.12:0/4130736172 shutdown_connections 2024-04-15T15:23:34.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.858+0000 7f4f5ea63640 1 -- 172.21.15.12:0/4130736172 wait complete. 2024-04-15T15:23:34.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.860+0000 7f4f5ea63640 1 Processor -- start 2024-04-15T15:23:34.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.860+0000 7f4f5ea63640 1 -- start start 2024-04-15T15:23:34.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5ea63640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f5813c020 0x7f4f5814e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5ea63640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f5814d4d0 0x7f4f58165880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5ea63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f581585a0 0x7f4f58158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5ea63640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4f580afe20 con 0x7f4f581585a0 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5ea63640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4f580729c0 con 0x7f4f5814d4d0 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5ea63640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f4f581666d0 con 0x7f4f5813c020 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f581585a0 0x7f4f58158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f581585a0 0x7f4f58158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53720/0 (socket says 172.21.15.12:53720) 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 -- 172.21.15.12:0/3377607756 learned_addr learned my addr 172.21.15.12:0/3377607756 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f57fff640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f5813c020 0x7f4f5814e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f577fe640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f5814d4d0 0x7f4f58165880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 -- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f5813c020 msgr2=0x7f4f5814e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f5813c020 0x7f4f5814e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 -- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f5814d4d0 msgr2=0x7f4f58165880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f5814d4d0 0x7f4f58165880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 -- 172.21.15.12:0/3377607756 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f58164500 con 0x7f4f581585a0 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.861+0000 7f4f5d000640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f581585a0 0x7f4f58158a50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4f58071e60 tx=0x7f4f5000f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.862+0000 7f4f54ff9640 1 -- 172.21.15.12:0/3377607756 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f50016620 con 0x7f4f581585a0 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.862+0000 7f4f54ff9640 1 -- 172.21.15.12:0/3377607756 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5001b070 con 0x7f4f581585a0 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.862+0000 7f4f54ff9640 1 -- 172.21.15.12:0/3377607756 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f500169e0 con 0x7f4f581585a0 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.862+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f581647e0 con 0x7f4f581585a0 2024-04-15T15:23:34.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.862+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f58164da0 con 0x7f4f581585a0 2024-04-15T15:23:34.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.863+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f580690f0 con 0x7f4f581585a0 2024-04-15T15:23:34.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.864+0000 7f4f54ff9640 1 -- 172.21.15.12:0/3377607756 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4f50005d10 con 0x7f4f581585a0 2024-04-15T15:23:34.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.864+0000 7f4f54ff9640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f4406c9b0 0x7f4f4406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:34.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.864+0000 7f4f54ff9640 1 -- 172.21.15.12:0/3377607756 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f4f5008c7a0 con 0x7f4f581585a0 2024-04-15T15:23:34.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.864+0000 7f4f57fff640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f4406c9b0 0x7f4f4406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:34.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.865+0000 7f4f57fff640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f4406c9b0 0x7f4f4406ee70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009970 tx=0x7f4f48008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:34.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:34.869+0000 7f4f54ff9640 1 -- 172.21.15.12:0/3377607756 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f50025020 con 0x7f4f581585a0 2024-04-15T15:23:35.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.052+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) -- 0x7f4f580b3fd0 con 0x7f4f581585a0 2024-04-15T15:23:35.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.055+0000 7f4f54ff9640 1 -- 172.21.15.12:0/3377607756 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v13) ==== 92+0+0 (secure 0 0 0) 0x7f4f5005ac80 con 0x7f4f581585a0 2024-04-15T15:23:35.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f4406c9b0 msgr2=0x7f4f4406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f4406c9b0 0x7f4f4406ee70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009970 tx=0x7f4f48008040 comp rx=0 tx=0).stop 2024-04-15T15:23:35.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f581585a0 msgr2=0x7f4f58158a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f581585a0 0x7f4f58158a50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4f58071e60 tx=0x7f4f5000f810 comp rx=0 tx=0).stop 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 shutdown_connections 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f581585a0 0x7f4f58158a50 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f4f4406c9b0 0x7f4f4406ee70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f5814d4d0 0x7f4f58165880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 --2- 172.21.15.12:0/3377607756 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f4f5813c020 0x7f4f5814e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 >> 172.21.15.12:0/3377607756 conn(0x7f4f5806be00 msgr2=0x7f4f5815ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 shutdown_connections 2024-04-15T15:23:35.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.058+0000 7f4f5ea63640 1 -- 172.21.15.12:0/3377607756 wait complete. 2024-04-15T15:23:35.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-15T15:23:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.176+0000 7fcce6be9640 1 Processor -- start 2024-04-15T15:23:35.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.176+0000 7fcce6be9640 1 -- start start 2024-04-15T15:23:35.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.176+0000 7fcce6be9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcce009a9d0 0x7fcce0160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.176+0000 7fcce6be9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcce015fb30 0x7fcce00d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.176+0000 7fcce6be9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcce0070720 con 0x7fcce00701f0 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.176+0000 7fcce6be9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcce009ba90 con 0x7fcce009a9d0 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.176+0000 7fcce6be9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fcce00d08c0 con 0x7fcce015fb30 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.177+0000 7fcce4985640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcce009a9d0 0x7fcce0160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.177+0000 7fcce4985640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcce009a9d0 0x7fcce0160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53726/0 (socket says 172.21.15.12:53726) 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.177+0000 7fcce4985640 1 -- 172.21.15.12:0/3526979106 learned_addr learned my addr 172.21.15.12:0/3526979106 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.177+0000 7fcce5186640 1 --2- 172.21.15.12:0/3526979106 >> v2:172.21.15.55:3300/0 conn(0x7fcce015fb30 0x7fcce00d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.177+0000 7fcce4985640 1 -- 172.21.15.12:0/3526979106 >> v2:172.21.15.55:3300/0 conn(0x7fcce015fb30 msgr2=0x7fcce00d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.177+0000 7fcce4985640 1 --2- 172.21.15.12:0/3526979106 >> v2:172.21.15.55:3300/0 conn(0x7fcce015fb30 0x7fcce00d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.177+0000 7fcce4985640 1 -- 172.21.15.12:0/3526979106 >> v1:172.21.15.12:6789/0 conn(0x7fcce00701f0 legacy=0x7fcce009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.177+0000 7fcce4985640 1 -- 172.21.15.12:0/3526979106 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce0070a90 con 0x7fcce009a9d0 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fcce4985640 1 --2- 172.21.15.12:0/3526979106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce009a9d0 0x7fcce0160420 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcce0073740 tx=0x7fccd402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f474a6385af1bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fccdf7fe640 1 -- 172.21.15.12:0/3526979106 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4034070 con 0x7fcce009a9d0 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fccdf7fe640 1 -- 172.21.15.12:0/3526979106 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccd4030860 con 0x7fcce009a9d0 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fccdf7fe640 1 -- 172.21.15.12:0/3526979106 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4030c30 con 0x7fcce009a9d0 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fcce6be9640 1 -- 172.21.15.12:0/3526979106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce009a9d0 msgr2=0x7fcce0160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fcce6be9640 1 --2- 172.21.15.12:0/3526979106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce009a9d0 0x7fcce0160420 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcce0073740 tx=0x7fccd402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fcce6be9640 1 -- 172.21.15.12:0/3526979106 shutdown_connections 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fcce6be9640 1 --2- 172.21.15.12:0/3526979106 >> v2:172.21.15.55:3300/0 conn(0x7fcce015fb30 0x7fcce00d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fcce6be9640 1 --2- 172.21.15.12:0/3526979106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce009a9d0 0x7fcce0160420 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fcce6be9640 1 -- 172.21.15.12:0/3526979106 >> 172.21.15.12:0/3526979106 conn(0x7fcce00d2460 msgr2=0x7fcce00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:35.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.178+0000 7fcce6be9640 1 -- 172.21.15.12:0/3526979106 shutdown_connections 2024-04-15T15:23:35.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.179+0000 7fcce6be9640 1 -- 172.21.15.12:0/3526979106 wait complete. 2024-04-15T15:23:35.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.180+0000 7fcce6be9640 1 Processor -- start 2024-04-15T15:23:35.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.180+0000 7fcce6be9640 1 -- start start 2024-04-15T15:23:35.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce6be9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce00701f0 0x7fcce015a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce6be9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce009a9d0 0x7fcce015d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce6be9640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcce015fb30 0x7fcce015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce6be9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcce015ffc0 con 0x7fcce00701f0 2024-04-15T15:23:35.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce6be9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcce0071050 con 0x7fcce009a9d0 2024-04-15T15:23:35.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce6be9640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fcce00b1c60 con 0x7fcce015fb30 2024-04-15T15:23:35.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce00701f0 0x7fcce015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce00701f0 0x7fcce015a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53738/0 (socket says 172.21.15.12:53738) 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 -- 172.21.15.12:0/1068296413 learned_addr learned my addr 172.21.15.12:0/1068296413 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce5186640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcce015fb30 0x7fcce015a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fccdffff640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce009a9d0 0x7fcce015d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 -- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcce015fb30 msgr2=0x7fcce015a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcce015fb30 0x7fcce015a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 -- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce009a9d0 msgr2=0x7fcce015d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce009a9d0 0x7fcce015d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 -- 172.21.15.12:0/1068296413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce015c0c0 con 0x7fcce00701f0 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fccdffff640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce009a9d0 0x7fcce015d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce4985640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce00701f0 0x7fcce015a920 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fccd402b9e0 tx=0x7fccd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fccdd7fa640 1 -- 172.21.15.12:0/1068296413 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4048070 con 0x7fcce00701f0 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fccdd7fa640 1 -- 172.21.15.12:0/1068296413 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccd4030660 con 0x7fcce00701f0 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce015b020 con 0x7fcce00701f0 2024-04-15T15:23:35.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fccdd7fa640 1 -- 172.21.15.12:0/1068296413 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4034040 con 0x7fcce00701f0 2024-04-15T15:23:35.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.181+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce015b5e0 con 0x7fcce00701f0 2024-04-15T15:23:35.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.182+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce00689e0 con 0x7fcce00701f0 2024-04-15T15:23:35.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.183+0000 7fccdd7fa640 1 -- 172.21.15.12:0/1068296413 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fccd404b070 con 0x7fcce00701f0 2024-04-15T15:23:35.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.184+0000 7fccdd7fa640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fccc406c980 0x7fccc406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.184+0000 7fccdd7fa640 1 -- 172.21.15.12:0/1068296413 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fccd40adbe0 con 0x7fcce00701f0 2024-04-15T15:23:35.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.184+0000 7fccdffff640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fccc406c980 0x7fccc406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.184+0000 7fccdffff640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fccc406c980 0x7fccc406ee40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcce00c9490 tx=0x7fcccc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.188+0000 7fccdd7fa640 1 -- 172.21.15.12:0/1068296413 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fccd407c030 con 0x7fcce00701f0 2024-04-15T15:23:35.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.373+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fcce00d3370 con 0x7fcce00701f0 2024-04-15T15:23:35.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.376+0000 7fccdd7fa640 1 -- 172.21.15.12:0/1068296413 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) ==== 104+0+0 (secure 0 0 0) 0x7fccd407b930 con 0x7fcce00701f0 2024-04-15T15:23:35.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.378+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fccc406c980 msgr2=0x7fccc406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.378+0000 7fcce6be9640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fccc406c980 0x7fccc406ee40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcce00c9490 tx=0x7fcccc009040 comp rx=0 tx=0).stop 2024-04-15T15:23:35.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.378+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce00701f0 msgr2=0x7fcce015a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.378+0000 7fcce6be9640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce00701f0 0x7fcce015a920 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fccd402b9e0 tx=0x7fccd4004730 comp rx=0 tx=0).stop 2024-04-15T15:23:35.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.379+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 shutdown_connections 2024-04-15T15:23:35.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.379+0000 7fcce6be9640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fcce015fb30 0x7fcce015a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.379+0000 7fcce6be9640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fccc406c980 0x7fccc406ee40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.379+0000 7fcce6be9640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce009a9d0 0x7fcce015d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.379+0000 7fcce6be9640 1 --2- 172.21.15.12:0/1068296413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce00701f0 0x7fcce015a920 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.379+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 >> 172.21.15.12:0/1068296413 conn(0x7fcce00d2460 msgr2=0x7fcce00b3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.379+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 shutdown_connections 2024-04-15T15:23:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.379+0000 7fcce6be9640 1 -- 172.21.15.12:0/1068296413 wait complete. 2024-04-15T15:23:35.391 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_cache_quiesce_splitauth 2024-04-15T15:23:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.496+0000 7f78f7b67640 1 Processor -- start 2024-04-15T15:23:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.496+0000 7f78f7b67640 1 -- start start 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f7b67640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f78f015e790 0x7f78f00b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f7b67640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f00d1cf0 0x7f78f00d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f7b67640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f015eb70 con 0x7f78f00b7be0 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f7b67640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f78f00d3790 con 0x7f78f015e790 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f7b67640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f78f00d45b0 con 0x7f78f00d1cf0 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f5102640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f78f00b7be0 0x7f78f00d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50318/0 (socket says 172.21.15.12:50318) 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f5102640 1 -- 172.21.15.12:0/4017634662 learned_addr learned my addr 172.21.15.12:0/4017634662 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f6104640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f00d1cf0 0x7f78f00d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f5903640 1 --2- 172.21.15.12:0/4017634662 >> v2:172.21.15.55:3300/0 conn(0x7f78f015e790 0x7f78f00b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1835831480 0 0) 0x7f78f015eb70 con 0x7f78f00b7be0 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78dc003520 con 0x7f78f00b7be0 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1542771847 0 0) 0x7f78dc003520 con 0x7f78f00b7be0 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f00d1cf0 msgr2=0x7f78f00d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f00d1cf0 0x7f78f00d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 >> v2:172.21.15.55:3300/0 conn(0x7f78f015e790 msgr2=0x7f78f00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 --2- 172.21.15.12:0/4017634662 >> v2:172.21.15.55:3300/0 conn(0x7f78f015e790 0x7f78f00b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f00d3c80 con 0x7f78f00b7be0 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f78e0001690 con 0x7f78f00b7be0 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 >> v1:172.21.15.12:6789/0 conn(0x7f78f00b7be0 legacy=0x7f78f00d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78dc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc004800 0x7f78dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.497+0000 7f78f4901640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f00d1cf0 0x7f78dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78f00d3790 con 0x7f78dc0039b0 2024-04-15T15:23:35.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f78f00d45b0 con 0x7f78dc004800 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f78dc00baf0 con 0x7f78f00d1cf0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f5903640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78dc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f5102640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc004800 0x7f78dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f6104640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f00d1cf0 0x7f78dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f5903640 1 -- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f00d1cf0 msgr2=0x7f78dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f5903640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f00d1cf0 0x7f78dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f5903640 1 -- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc004800 msgr2=0x7f78dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f5903640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc004800 0x7f78dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.498+0000 7f78f5903640 1 -- 172.21.15.12:0/4017634662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78dc003520 con 0x7f78dc0039b0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f5903640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78dc003df0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009a90 tx=0x7f78dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582f19bc17ca0565 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec039070 con 0x7f78dc0039b0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ec02c6b0 con 0x7f78dc0039b0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f4901640 1 -- 172.21.15.12:0/4017634662 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec009690 con 0x7f78dc0039b0 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f7b67640 1 -- 172.21.15.12:0/4017634662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 msgr2=0x7f78dc003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f7b67640 1 --2- 172.21.15.12:0/4017634662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78dc003df0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009a90 tx=0x7f78dc005830 comp rx=0 tx=0).stop 2024-04-15T15:23:35.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f6104640 1 -- 172.21.15.12:0/4017634662 reap_dead start 2024-04-15T15:23:35.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f7b67640 1 -- 172.21.15.12:0/4017634662 shutdown_connections 2024-04-15T15:23:35.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.499+0000 7f78f7b67640 1 -- 172.21.15.12:0/4017634662 >> 172.21.15.12:0/4017634662 conn(0x7f78f0067b40 msgr2=0x7f78f0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:35.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.500+0000 7f78f7b67640 1 -- 172.21.15.12:0/4017634662 shutdown_connections 2024-04-15T15:23:35.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.500+0000 7f78f7b67640 1 -- 172.21.15.12:0/4017634662 wait complete. 2024-04-15T15:23:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.501+0000 7f78f7b67640 1 Processor -- start 2024-04-15T15:23:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.501+0000 7f78f7b67640 1 -- start start 2024-04-15T15:23:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.501+0000 7f78f7b67640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78f014f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.501+0000 7f78f7b67640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78f014d030 0x7f78f014d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.501+0000 7f78f7b67640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f014c140 0x7f78f014c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.501+0000 7f78f7b67640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78f00d4a40 con 0x7f78dc0039b0 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f7b67640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f78f00d4240 con 0x7f78f014d030 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f7b67640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f78f00d4bc0 con 0x7f78f014c140 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78f014f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78f014f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53756/0 (socket says 172.21.15.12:53756) 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 -- 172.21.15.12:0/2475529525 learned_addr learned my addr 172.21.15.12:0/2475529525 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 -- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f014c140 msgr2=0x7f78f014c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f014c140 0x7f78f014c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 -- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78f014d030 msgr2=0x7f78f014d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78f014d030 0x7f78f014d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 -- 172.21.15.12:0/2475529525 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f014dda0 con 0x7f78dc0039b0 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f5903640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78f014f250 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f78ec000c00 tx=0x7f78ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78e67fc640 1 -- 172.21.15.12:0/2475529525 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec039050 con 0x7f78dc0039b0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78e67fc640 1 -- 172.21.15.12:0/2475529525 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ec045070 con 0x7f78dc0039b0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f014e080 con 0x7f78dc0039b0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78e67fc640 1 -- 172.21.15.12:0/2475529525 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec0044e0 con 0x7f78dc0039b0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.502+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f014e640 con 0x7f78dc0039b0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.503+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f015fd80 con 0x7f78dc0039b0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.504+0000 7f78e67fc640 1 -- 172.21.15.12:0/2475529525 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f78ec003a50 con 0x7f78dc0039b0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.504+0000 7f78e67fc640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78d406c9b0 0x7f78d406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.504+0000 7f78e67fc640 1 -- 172.21.15.12:0/2475529525 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f78ec0ada10 con 0x7f78dc0039b0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.504+0000 7f78f5102640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78d406c9b0 0x7f78d406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.505+0000 7f78f5102640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78d406c9b0 0x7f78d406ee70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f78e0001d10 tx=0x7f78e0001c60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.508+0000 7f78e67fc640 1 -- 172.21.15.12:0/2475529525 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78f015fd80 con 0x7f78dc0039b0 2024-04-15T15:23:35.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.692+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"} v 0) -- 0x7f78f00d4fa0 con 0x7f78dc0039b0 2024-04-15T15:23:35.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.695+0000 7f78e67fc640 1 -- 172.21.15.12:0/2475529525 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"}]=0 v15) ==== 110+0+0 (secure 0 0 0) 0x7f78ec04f020 con 0x7f78dc0039b0 2024-04-15T15:23:35.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78d406c9b0 msgr2=0x7f78d406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78d406c9b0 0x7f78d406ee70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f78e0001d10 tx=0x7f78e0001c60 comp rx=0 tx=0).stop 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 msgr2=0x7f78f014f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78f014f250 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f78ec000c00 tx=0x7f78ec004730 comp rx=0 tx=0).stop 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 shutdown_connections 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78f014c140 0x7f78f014c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78d406c9b0 0x7f78d406ee70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78f014d030 0x7f78f014d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 --2- 172.21.15.12:0/2475529525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78dc0039b0 0x7f78f014f250 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 >> 172.21.15.12:0/2475529525 conn(0x7f78f0067b40 msgr2=0x7f78f00d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 shutdown_connections 2024-04-15T15:23:35.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.697+0000 7f78f7b67640 1 -- 172.21.15.12:0/2475529525 wait complete. 2024-04-15T15:23:35.709 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-15T15:23:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.813+0000 7f0b42904640 1 Processor -- start 2024-04-15T15:23:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.813+0000 7f0b42904640 1 -- start start 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.814+0000 7f0b42904640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.814+0000 7f0b42904640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0b3c09b0e0 0x7f0b3c0cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.814+0000 7f0b42904640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b3c06ff30 con 0x7f0b3c071d90 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.814+0000 7f0b42904640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0b3c070820 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.814+0000 7f0b42904640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0b3c0cecc0 con 0x7f0b3c0ce8e0 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b3bfff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b40ea1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0b3c09b0e0 0x7f0b3c0cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b33fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0b3c071d90 0x7f0b3c09a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:43474/0 (socket says 172.21.15.12:43474) 2024-04-15T15:23:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b33fff640 1 -- 172.21.15.12:0/2674154303 learned_addr learned my addr 172.21.15.12:0/2674154303 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:35.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b40ea1640 1 -- 172.21.15.12:0/2674154303 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 msgr2=0x7f0b3c070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:35.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b40ea1640 1 --2- 172.21.15.12:0/2674154303 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b40ea1640 1 -- 172.21.15.12:0/2674154303 >> v1:172.21.15.55:6789/0 conn(0x7f0b3c071d90 legacy=0x7f0b3c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b40ea1640 1 -- 172.21.15.12:0/2674154303 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3c09b9d0 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.815+0000 7f0b3bfff640 1 --2- 172.21.15.12:0/2674154303 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.816+0000 7f0b40ea1640 1 --2- 172.21.15.12:0/2674154303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c0cce80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009d90 tx=0x7f0b3402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a18a7e0e108c4cf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.816+0000 7f0b3b7fe640 1 -- 172.21.15.12:0/2674154303 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34039070 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.816+0000 7f0b3b7fe640 1 -- 172.21.15.12:0/2674154303 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b3402cbc0 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.816+0000 7f0b3b7fe640 1 -- 172.21.15.12:0/2674154303 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34034440 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.817+0000 7f0b42904640 1 -- 172.21.15.12:0/2674154303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c09b0e0 msgr2=0x7f0b3c0cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.817+0000 7f0b42904640 1 --2- 172.21.15.12:0/2674154303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c0cce80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009d90 tx=0x7f0b3402be10 comp rx=0 tx=0).stop 2024-04-15T15:23:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.817+0000 7f0b42904640 1 -- 172.21.15.12:0/2674154303 shutdown_connections 2024-04-15T15:23:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.817+0000 7f0b42904640 1 --2- 172.21.15.12:0/2674154303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c0cce80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.817+0000 7f0b42904640 1 --2- 172.21.15.12:0/2674154303 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.817+0000 7f0b42904640 1 -- 172.21.15.12:0/2674154303 >> 172.21.15.12:0/2674154303 conn(0x7f0b3c0b3f40 msgr2=0x7f0b3c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:35.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.817+0000 7f0b42904640 1 -- 172.21.15.12:0/2674154303 shutdown_connections 2024-04-15T15:23:35.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.817+0000 7f0b42904640 1 -- 172.21.15.12:0/2674154303 wait complete. 2024-04-15T15:23:35.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.818+0000 7f0b42904640 1 Processor -- start 2024-04-15T15:23:35.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.818+0000 7f0b42904640 1 -- start start 2024-04-15T15:23:35.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b42904640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c071d90 0x7f0b3c1d8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b42904640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c09d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b42904640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b42904640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c0704b0 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b42904640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c0701d0 con 0x7f0b3c071d90 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b42904640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0b3c071970 con 0x7f0b3c0ce8e0 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c09d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c09d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53760/0 (socket says 172.21.15.12:53760) 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 -- 172.21.15.12:0/2915510638 learned_addr learned my addr 172.21.15.12:0/2915510638 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b3bfff640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c071d90 0x7f0b3c1d8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b40ea1640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 -- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 msgr2=0x7f0b3c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 -- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c071d90 msgr2=0x7f0b3c1d8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c071d90 0x7f0b3c1d8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 -- 172.21.15.12:0/2915510638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3c1e47f0 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.819+0000 7f0b33fff640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c09d080 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400c3c0 tx=0x7f0b2400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.820+0000 7f0b397fa640 1 -- 172.21.15.12:0/2915510638 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24017070 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.820+0000 7f0b397fa640 1 -- 172.21.15.12:0/2915510638 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b24005ce0 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.820+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3c1e4ad0 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.820+0000 7f0b397fa640 1 -- 172.21.15.12:0/2915510638 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24007310 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.820+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3c1e5090 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.821+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b3c0689e0 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.821+0000 7f0b397fa640 1 -- 172.21.15.12:0/2915510638 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0b2401a070 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.821+0000 7f0b397fa640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b1c06c9b0 0x7f0b1c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.822+0000 7f0b397fa640 1 -- 172.21.15.12:0/2915510638 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f0b2408acd0 con 0x7f0b3c09b0e0 2024-04-15T15:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.822+0000 7f0b3bfff640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b1c06c9b0 0x7f0b1c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.822+0000 7f0b3bfff640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b1c06c9b0 0x7f0b1c06ee70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009560 tx=0x7f0b2c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:35.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:35.826+0000 7f0b397fa640 1 -- 172.21.15.12:0/2915510638 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b24058be0 con 0x7f0b3c09b0e0 2024-04-15T15:23:36.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.084+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7f0b3c0b4ca0 con 0x7f0b3c09b0e0 2024-04-15T15:23:36.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.723+0000 7f0b397fa640 1 -- 172.21.15.12:0/2915510638 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v156) ==== 137+0+0 (secure 0 0 0) 0x7f0b2405cee0 con 0x7f0b3c09b0e0 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b1c06c9b0 msgr2=0x7f0b1c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b1c06c9b0 0x7f0b1c06ee70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009560 tx=0x7f0b2c009190 comp rx=0 tx=0).stop 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b3c09b0e0 msgr2=0x7f0b3c09d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c09d080 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400c3c0 tx=0x7f0b2400c880 comp rx=0 tx=0).stop 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 shutdown_connections 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0b3c0ce8e0 0x7f0b3c0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b3c09b0e0 0x7f0b3c09d080 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0b1c06c9b0 0x7f0b1c06ee70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 --2- 172.21.15.12:0/2915510638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b3c071d90 0x7f0b3c1d8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 >> 172.21.15.12:0/2915510638 conn(0x7f0b3c0b3f40 msgr2=0x7f0b3c0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 shutdown_connections 2024-04-15T15:23:36.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:36.726+0000 7f0b42904640 1 -- 172.21.15.12:0/2915510638 wait complete. 2024-04-15T15:23:36.742 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. ... FAIL 2024-04-15T15:23:36.742 INFO:tasks.cephfs_test_runner: 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner:FAIL: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 541, in test_quiesce_blocked 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner: self._verify_quiesce(root=self.subvolume) 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 138, in _verify_quiesce 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner: return self._verify_quiesce_wrapped(rank, status, root, root_inode, ops, cache, splitauth) 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 272, in _verify_quiesce_wrapped 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner: self.assertFalse(lock_type.startswith("i")) 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner:AssertionError: True is not false 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner: 2024-04-15T15:23:36.743 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner:Ran 1 test in 98.333s 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner: 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner: 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner:FAIL: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 541, in test_quiesce_blocked 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner: self._verify_quiesce(root=self.subvolume) 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 138, in _verify_quiesce 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner: return self._verify_quiesce_wrapped(rank, status, root, root_inode, ops, cache, splitauth) 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 272, in _verify_quiesce_wrapped 2024-04-15T15:23:36.744 INFO:tasks.cephfs_test_runner: self.assertFalse(lock_type.startswith("i")) 2024-04-15T15:23:36.745 INFO:tasks.cephfs_test_runner:AssertionError: True is not false 2024-04-15T15:23:36.745 INFO:tasks.cephfs_test_runner: 2024-04-15T15:23:36.745 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:23:36.981 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=23c3ed57813f4ed6818c345f16c1a286 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:23:36.983 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-15T15:23:36.992 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-15T15:23:37.027 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-15T15:23:37.027 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:23:37.027 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:37.049 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20669 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:23:37.050 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-15T15:23:37.050 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:37.108 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20669 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:23:37.109 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:23:37.109 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:37.128 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14740 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:23:37.128 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-15T15:23:37.128 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:23:37.187 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14740 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:23:37.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:37.188 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-15T15:23:37.206 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:23:37.206 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:23:37.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:37.207 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:23:37.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:37.279 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:23:37.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:37.352 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-04-15T15:23:37.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:37.458 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-04-15T15:23:37.477 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:23:37.477 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:23:37.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:37.478 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:23:37.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:37.551 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:23:37.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:23:37.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link delete ceph-brx 2024-04-15T15:23:37.720 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-15T15:23:37.730 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-15T15:23:37.731 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:23:37.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.842+0000 7efff0460640 1 Processor -- start 2024-04-15T15:23:37.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.842+0000 7efff0460640 1 -- start start 2024-04-15T15:23:37.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7efff0460640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7effe8072d80 0x7effe8071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:37.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7efff0460640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe815db40 con 0x7effe80af3e0 2024-04-15T15:23:37.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7efff0460640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe815cfc0 con 0x7effe80d0da0 2024-04-15T15:23:37.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7efff0460640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7effe815d2a0 con 0x7effe8072d80 2024-04-15T15:23:37.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7effed9fb640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7effe80af3e0 0x7effe80d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50320/0 (socket says 172.21.15.12:50320) 2024-04-15T15:23:37.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7effed9fb640 1 -- 172.21.15.12:0/3271284227 learned_addr learned my addr 172.21.15.12:0/3271284227 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:37.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7effee1fc640 1 --2- 172.21.15.12:0/3271284227 >> v2:172.21.15.55:3300/0 conn(0x7effe8072d80 0x7effe8071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:37.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3181348092 0 0) 0x7effe815db40 con 0x7effe80af3e0 2024-04-15T15:23:37.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.843+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effd4003520 con 0x7effe80af3e0 2024-04-15T15:23:37.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 966400442 0 0) 0x7effd4003520 con 0x7effe80af3e0 2024-04-15T15:23:37.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 >> v2:172.21.15.55:3300/0 conn(0x7effe8072d80 msgr2=0x7effe8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:37.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 --2- 172.21.15.12:0/3271284227 >> v2:172.21.15.55:3300/0 conn(0x7effe8072d80 0x7effe8071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:37.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 >> v1:172.21.15.55:6789/0 conn(0x7effe80d0da0 legacy=0x7effe806e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:37.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe815d580 con 0x7effe80af3e0 2024-04-15T15:23:37.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7effd8001670 con 0x7effe80af3e0 2024-04-15T15:23:37.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 >> v1:172.21.15.12:6789/0 conn(0x7effe80af3e0 legacy=0x7effe80d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:37.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:37.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effd4004800 0x7effd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:37.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effd40051c0 0x7effd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:37.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7effe815d2a0 con 0x7effd40051c0 2024-04-15T15:23:37.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7effd400c160 con 0x7effd40039b0 2024-04-15T15:23:37.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7effd400c360 con 0x7effd4004800 2024-04-15T15:23:37.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee1fc640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:37.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee9fd640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effd40051c0 0x7effd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:37.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effed9fb640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effd4004800 0x7effd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:37.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee1fc640 1 -- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effd4004800 msgr2=0x7effd4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:37.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee1fc640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effd4004800 0x7effd4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:37.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee1fc640 1 -- 172.21.15.12:0/3271284227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effd40051c0 msgr2=0x7effd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:37.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee1fc640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effd40051c0 0x7effd400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:37.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee1fc640 1 -- 172.21.15.12:0/3271284227 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd4003520 con 0x7effd40039b0 2024-04-15T15:23:37.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee9fd640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effd40051c0 0x7effd400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:37.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.844+0000 7effee9fd640 1 -- 172.21.15.12:0/3271284227 reap_dead start 2024-04-15T15:23:37.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.845+0000 7effee1fc640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effd4003dd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7effe4009ac0 tx=0x7effd4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a55a73e915908e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:37.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.845+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4039070 con 0x7effd40039b0 2024-04-15T15:23:37.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.845+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe402c6b0 con 0x7effd40039b0 2024-04-15T15:23:37.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.845+0000 7effed1fa640 1 -- 172.21.15.12:0/3271284227 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4009690 con 0x7effd40039b0 2024-04-15T15:23:37.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.846+0000 7efff0460640 1 -- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 msgr2=0x7effd4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:37.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.846+0000 7efff0460640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effd4003dd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7effe4009ac0 tx=0x7effd4005830 comp rx=0 tx=0).stop 2024-04-15T15:23:37.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.846+0000 7efff0460640 1 -- 172.21.15.12:0/3271284227 shutdown_connections 2024-04-15T15:23:37.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.846+0000 7efff0460640 1 --2- 172.21.15.12:0/3271284227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effd4003dd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:37.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.846+0000 7efff0460640 1 -- 172.21.15.12:0/3271284227 >> 172.21.15.12:0/3271284227 conn(0x7effe806be00 msgr2=0x7effe806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:37.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.846+0000 7efff0460640 1 -- 172.21.15.12:0/3271284227 shutdown_connections 2024-04-15T15:23:37.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.846+0000 7efff0460640 1 -- 172.21.15.12:0/3271284227 wait complete. 2024-04-15T15:23:37.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.847+0000 7efff0460640 1 Processor -- start 2024-04-15T15:23:37.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.847+0000 7efff0460640 1 -- start start 2024-04-15T15:23:37.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7efff0460640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effe80b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:37.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7efff0460640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effe80b1e40 0x7effe80b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:37.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7efff0460640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effe80cf230 0x7effe80cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:37.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7efff0460640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7effe815cd10 con 0x7effe80cf230 2024-04-15T15:23:37.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7efff0460640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7effe815cb90 con 0x7effd40039b0 2024-04-15T15:23:37.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7efff0460640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7effe80af840 con 0x7effe80b1e40 2024-04-15T15:23:37.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effe80cf230 0x7effe80cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:37.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effe80cf230 0x7effe80cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53782/0 (socket says 172.21.15.12:53782) 2024-04-15T15:23:37.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 -- 172.21.15.12:0/2243321469 learned_addr learned my addr 172.21.15.12:0/2243321469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:37.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee1fc640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effe80b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:37.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effed9fb640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effe80b1e40 0x7effe80b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:37.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 -- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effe80b1e40 msgr2=0x7effe80b22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:37.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effe80b1e40 0x7effe80b22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:37.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 -- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 msgr2=0x7effe80b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:37.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effe80b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:37.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 -- 172.21.15.12:0/2243321469 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe80cb180 con 0x7effe80cf230 2024-04-15T15:23:37.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.848+0000 7effee9fd640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effe80cf230 0x7effe80cf6b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7effe81600e0 tx=0x7effe000feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:37.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.849+0000 7effdeffd640 1 -- 172.21.15.12:0/2243321469 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe001a070 con 0x7effe80cf230 2024-04-15T15:23:37.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.849+0000 7effdeffd640 1 -- 172.21.15.12:0/2243321469 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe0006d60 con 0x7effe80cf230 2024-04-15T15:23:37.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.849+0000 7effdeffd640 1 -- 172.21.15.12:0/2243321469 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0001a70 con 0x7effe80cf230 2024-04-15T15:23:37.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.849+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe80cfbf0 con 0x7effe80cf230 2024-04-15T15:23:37.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.849+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe80cb6e0 con 0x7effe80cf230 2024-04-15T15:23:37.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.850+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe80689e0 con 0x7effe80cf230 2024-04-15T15:23:37.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.850+0000 7effdeffd640 1 -- 172.21.15.12:0/2243321469 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7effe0003b90 con 0x7effe80cf230 2024-04-15T15:23:37.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.851+0000 7effdeffd640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7effcc06c980 0x7effcc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:37.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.851+0000 7effdeffd640 1 -- 172.21.15.12:0/2243321469 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7effe008ae20 con 0x7effe80cf230 2024-04-15T15:23:37.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.851+0000 7effee1fc640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7effcc06c980 0x7effcc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:37.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.851+0000 7effee1fc640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7effcc06c980 0x7effcc06ee40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7effe4036000 tx=0x7effe4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:37.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:37.855+0000 7effdeffd640 1 -- 172.21.15.12:0/2243321469 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effe80cfbf0 con 0x7effe80cf230 2024-04-15T15:23:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.035+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7effe8071db0 con 0x7effcc06c980 2024-04-15T15:23:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.036+0000 7effdeffd640 1 -- 172.21.15.12:0/2243321469 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13408 (secure 0 0 0) 0x7effe8071db0 con 0x7effcc06c980 2024-04-15T15:23:38.039 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:38.039 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:23:38.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7effcc06c980 msgr2=0x7effcc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7effcc06c980 0x7effcc06ee40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7effe4036000 tx=0x7effe4030000 comp rx=0 tx=0).stop 2024-04-15T15:23:38.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effe80cf230 msgr2=0x7effe80cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effe80cf230 0x7effe80cf6b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7effe81600e0 tx=0x7effe000feb0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 shutdown_connections 2024-04-15T15:23:38.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effe80cf230 0x7effe80cf6b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7effe80b1e40 0x7effe80b22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7effcc06c980 0x7effcc06ee40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 --2- 172.21.15.12:0/2243321469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effd40039b0 0x7effe80b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.038+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 >> 172.21.15.12:0/2243321469 conn(0x7effe806be00 msgr2=0x7effe815bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:38.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.039+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 shutdown_connections 2024-04-15T15:23:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.039+0000 7efff0460640 1 -- 172.21.15.12:0/2243321469 wait complete. 2024-04-15T15:23:38.051 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-04-15T15:23:36.623930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237968,"kb_used_data":5180,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736752,"statfs":{"total":773094113280,"available":772850434048,"internally_reserved":0,"allocated":5304320,"data_stored":3036770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62581,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-130,"num_write":-96,"num_write_kb":-97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001348"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":213,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-15T15:23:32.556232+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:23:32.556232+0000","last_peered":"2024-04-15T15:23:32.556232+0000","last_clean":"2024-04-15T15:23:32.556232+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:23:32.556232+0000","last_undegraded":"2024-04-15T15:23:32.556232+0000","last_fullsized":"2024-04-15T15:23:32.556232+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1167360,"data_stored":871673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":214945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":495616,"data_stored":214342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":214834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":569344,"data_stored":281136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":503808,"data_stored":220888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":491520,"data_stored":214320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1085440,"data_stored":804632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:23:38.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:23:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.154+0000 7fc493c69640 1 Processor -- start 2024-04-15T15:23:38.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.155+0000 7fc493c69640 1 -- start start 2024-04-15T15:23:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.155+0000 7fc493c69640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc48c0b49d0 0x7fc48c160af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.155+0000 7fc493c69640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.155+0000 7fc493c69640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc48c0d14b0 con 0x7fc48c09ac40 2024-04-15T15:23:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.155+0000 7fc493c69640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc48c0b1b70 con 0x7fc48c0b49d0 2024-04-15T15:23:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.155+0000 7fc493c69640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc48c0cd550 con 0x7fc48c09b020 2024-04-15T15:23:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc492206640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc491204640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc48c0b49d0 0x7fc48c160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc491a05640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc48c09ac40 0x7fc48c0d0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:43498/0 (socket says 172.21.15.12:43498) 2024-04-15T15:23:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc492206640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3301/0 says I am v2:172.21.15.12:43904/0 (socket says 172.21.15.12:43904) 2024-04-15T15:23:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc491a05640 1 -- 172.21.15.12:0/2818675724 learned_addr learned my addr 172.21.15.12:0/2818675724 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc491204640 1 -- 172.21.15.12:0/2818675724 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 msgr2=0x7fc48c0ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc491204640 1 --2- 172.21.15.12:0/2818675724 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc491204640 1 -- 172.21.15.12:0/2818675724 >> v1:172.21.15.55:6789/0 conn(0x7fc48c09ac40 legacy=0x7fc48c0d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc491204640 1 -- 172.21.15.12:0/2818675724 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48c0cd830 con 0x7fc48c0b49d0 2024-04-15T15:23:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.156+0000 7fc492206640 1 --2- 172.21.15.12:0/2818675724 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc491204640 1 --2- 172.21.15.12:0/2818675724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c0b49d0 0x7fc48c160af0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc4800097a0 tx=0x7fc48002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac554578f335bdf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc490a03640 1 -- 172.21.15.12:0/2818675724 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480034070 con 0x7fc48c0b49d0 2024-04-15T15:23:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc490a03640 1 -- 172.21.15.12:0/2818675724 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc480030610 con 0x7fc48c0b49d0 2024-04-15T15:23:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc490a03640 1 -- 172.21.15.12:0/2818675724 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4800309e0 con 0x7fc48c0b49d0 2024-04-15T15:23:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc493c69640 1 -- 172.21.15.12:0/2818675724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c0b49d0 msgr2=0x7fc48c160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc493c69640 1 --2- 172.21.15.12:0/2818675724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c0b49d0 0x7fc48c160af0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc4800097a0 tx=0x7fc48002b960 comp rx=0 tx=0).stop 2024-04-15T15:23:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc493c69640 1 -- 172.21.15.12:0/2818675724 shutdown_connections 2024-04-15T15:23:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc493c69640 1 --2- 172.21.15.12:0/2818675724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c0b49d0 0x7fc48c160af0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc493c69640 1 --2- 172.21.15.12:0/2818675724 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.157+0000 7fc493c69640 1 -- 172.21.15.12:0/2818675724 >> 172.21.15.12:0/2818675724 conn(0x7fc48c0b3eb0 msgr2=0x7fc48c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.158+0000 7fc493c69640 1 -- 172.21.15.12:0/2818675724 shutdown_connections 2024-04-15T15:23:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.158+0000 7fc493c69640 1 -- 172.21.15.12:0/2818675724 wait complete. 2024-04-15T15:23:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.159+0000 7fc493c69640 1 Processor -- start 2024-04-15T15:23:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.159+0000 7fc493c69640 1 -- start start 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.159+0000 7fc493c69640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c09ac40 0x7fc48c0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.159+0000 7fc493c69640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.159+0000 7fc493c69640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc48c0b49d0 0x7fc48c1e7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.159+0000 7fc493c69640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc48c0ce570 con 0x7fc48c09ac40 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.159+0000 7fc493c69640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc48c161030 con 0x7fc48c0b49d0 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.159+0000 7fc493c69640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fc48c0ce6f0 con 0x7fc48c09b020 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c09ac40 0x7fc48c0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c09ac40 0x7fc48c0ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53804/0 (socket says 172.21.15.12:53804) 2024-04-15T15:23:38.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 -- 172.21.15.12:0/1184328598 learned_addr learned my addr 172.21.15.12:0/1184328598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:38.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc492206640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc48c0b49d0 0x7fc48c1e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491204640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 -- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 msgr2=0x7fc48c0cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 -- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc48c0b49d0 msgr2=0x7fc48c1e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc48c0b49d0 0x7fc48c1e7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 -- 172.21.15.12:0/1184328598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48c1e7570 con 0x7fc48c09ac40 2024-04-15T15:23:38.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc491a05640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c09ac40 0x7fc48c0ca9d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc48400c3c0 tx=0x7fc48400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc48a7fc640 1 -- 172.21.15.12:0/1184328598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484017070 con 0x7fc48c09ac40 2024-04-15T15:23:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc48a7fc640 1 -- 172.21.15.12:0/1184328598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc484005ce0 con 0x7fc48c09ac40 2024-04-15T15:23:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48c0cb030 con 0x7fc48c09ac40 2024-04-15T15:23:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc48a7fc640 1 -- 172.21.15.12:0/1184328598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4840072f0 con 0x7fc48c09ac40 2024-04-15T15:23:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.160+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc48c0cb870 con 0x7fc48c09ac40 2024-04-15T15:23:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.161+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc48c0689e0 con 0x7fc48c09ac40 2024-04-15T15:23:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.162+0000 7fc48a7fc640 1 -- 172.21.15.12:0/1184328598 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc48401a070 con 0x7fc48c09ac40 2024-04-15T15:23:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.162+0000 7fc48a7fc640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc47806c980 0x7fc47806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.162+0000 7fc48a7fc640 1 -- 172.21.15.12:0/1184328598 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fc48408ad80 con 0x7fc48c09ac40 2024-04-15T15:23:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.163+0000 7fc491204640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc47806c980 0x7fc47806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.163+0000 7fc491204640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc47806c980 0x7fc47806ee40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc480036000 tx=0x7fc48003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.167+0000 7fc48a7fc640 1 -- 172.21.15.12:0/1184328598 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc48c0cb030 con 0x7fc48c09ac40 2024-04-15T15:23:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.346+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc48c09b500 con 0x7fc47806c980 2024-04-15T15:23:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.348+0000 7fc48a7fc640 1 -- 172.21.15.12:0/1184328598 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13408 (secure 0 0 0) 0x7fc48c09b500 con 0x7fc47806c980 2024-04-15T15:23:38.349 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:23:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc47806c980 msgr2=0x7fc47806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc47806c980 0x7fc47806ee40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc480036000 tx=0x7fc48003b040 comp rx=0 tx=0).stop 2024-04-15T15:23:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c09ac40 msgr2=0x7fc48c0ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c09ac40 0x7fc48c0ca9d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc48400c3c0 tx=0x7fc48400c880 comp rx=0 tx=0).stop 2024-04-15T15:23:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 shutdown_connections 2024-04-15T15:23:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fc47806c980 0x7fc47806ee40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc48c0b49d0 0x7fc48c1e7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fc48c09b020 0x7fc48c0cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 --2- 172.21.15.12:0/1184328598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc48c09ac40 0x7fc48c0ca9d0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 >> 172.21.15.12:0/1184328598 conn(0x7fc48c0b3eb0 msgr2=0x7fc48c0b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 shutdown_connections 2024-04-15T15:23:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.350+0000 7fc493c69640 1 -- 172.21.15.12:0/1184328598 wait complete. 2024-04-15T15:23:38.362 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-04-15T15:23:36.623930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237968,"kb_used_data":5180,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736752,"statfs":{"total":773094113280,"available":772850434048,"internally_reserved":0,"allocated":5304320,"data_stored":3036770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62581,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-130,"num_write":-96,"num_write_kb":-97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001348"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":213,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-15T15:23:32.556232+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:23:32.556232+0000","last_peered":"2024-04-15T15:23:32.556232+0000","last_clean":"2024-04-15T15:23:32.556232+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:23:32.556232+0000","last_undegraded":"2024-04-15T15:23:32.556232+0000","last_fullsized":"2024-04-15T15:23:32.556232+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1167360,"data_stored":871673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":214945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":495616,"data_stored":214342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":214834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":569344,"data_stored":281136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":503808,"data_stored":220888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":491520,"data_stored":214320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1085440,"data_stored":804632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:23:38.362 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-15T15:23:38.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:23:38.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.466+0000 7fd995e47640 1 Processor -- start 2024-04-15T15:23:38.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.466+0000 7fd995e47640 1 -- start start 2024-04-15T15:23:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd995e47640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9900cd000 0x7fd9900d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd995e47640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd99009ab40 0x7fd99009afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd995e47640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd99015dbd0 con 0x7fd9901602d0 2024-04-15T15:23:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd995e47640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd9900cd460 con 0x7fd9900cd000 2024-04-15T15:23:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd995e47640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd9900cd6a0 con 0x7fd99009ab40 2024-04-15T15:23:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd987fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9900cd000 0x7fd9900d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd987fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9900cd000 0x7fd9900d0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53820/0 (socket says 172.21.15.12:53820) 2024-04-15T15:23:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd987fff640 1 -- 172.21.15.12:0/622013050 learned_addr learned my addr 172.21.15.12:0/622013050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd98f7fe640 1 --1- 172.21.15.12:0/622013050 >> v1:172.21.15.12:6789/0 conn(0x7fd9901602d0 0x7fd9901606b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50328/0 (socket says 172.21.15.12:50328) 2024-04-15T15:23:38.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd98ffff640 1 --2- 172.21.15.12:0/622013050 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd99009ab40 0x7fd99009afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd987fff640 1 -- 172.21.15.12:0/622013050 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd99009ab40 msgr2=0x7fd99009afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd987fff640 1 --2- 172.21.15.12:0/622013050 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd99009ab40 0x7fd99009afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd987fff640 1 -- 172.21.15.12:0/622013050 >> v1:172.21.15.12:6789/0 conn(0x7fd9901602d0 legacy=0x7fd9901606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd987fff640 1 -- 172.21.15.12:0/622013050 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9900cd980 con 0x7fd9900cd000 2024-04-15T15:23:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.467+0000 7fd98ffff640 1 --2- 172.21.15.12:0/622013050 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd99009ab40 0x7fd99009afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.468+0000 7fd987fff640 1 --2- 172.21.15.12:0/622013050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9900cd000 0x7fd9900d0ef0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fd978009870 tx=0x7fd97802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219b93e453a711ad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.468+0000 7fd98effd640 1 -- 172.21.15.12:0/622013050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978034070 con 0x7fd9900cd000 2024-04-15T15:23:38.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.468+0000 7fd98effd640 1 -- 172.21.15.12:0/622013050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd978030840 con 0x7fd9900cd000 2024-04-15T15:23:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.468+0000 7fd98effd640 1 -- 172.21.15.12:0/622013050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978030c10 con 0x7fd9900cd000 2024-04-15T15:23:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.469+0000 7fd995e47640 1 -- 172.21.15.12:0/622013050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9900cd000 msgr2=0x7fd9900d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.469+0000 7fd995e47640 1 --2- 172.21.15.12:0/622013050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9900cd000 0x7fd9900d0ef0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fd978009870 tx=0x7fd97802b980 comp rx=0 tx=0).stop 2024-04-15T15:23:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.469+0000 7fd995e47640 1 -- 172.21.15.12:0/622013050 shutdown_connections 2024-04-15T15:23:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.469+0000 7fd995e47640 1 --2- 172.21.15.12:0/622013050 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd99009ab40 0x7fd99009afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.469+0000 7fd995e47640 1 --2- 172.21.15.12:0/622013050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9900cd000 0x7fd9900d0ef0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.469+0000 7fd995e47640 1 -- 172.21.15.12:0/622013050 >> 172.21.15.12:0/622013050 conn(0x7fd9900d2460 msgr2=0x7fd9900d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:38.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.469+0000 7fd995e47640 1 -- 172.21.15.12:0/622013050 shutdown_connections 2024-04-15T15:23:38.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.469+0000 7fd995e47640 1 -- 172.21.15.12:0/622013050 wait complete. 2024-04-15T15:23:38.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.470+0000 7fd995e47640 1 Processor -- start 2024-04-15T15:23:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.470+0000 7fd995e47640 1 -- start start 2024-04-15T15:23:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd995e47640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd99009ab40 0x7fd99015ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd995e47640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9900cd000 0x7fd9900b7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd995e47640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9901602d0 0x7fd9900af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd995e47640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd9900ce220 con 0x7fd9901602d0 2024-04-15T15:23:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd995e47640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd9900cdf40 con 0x7fd99009ab40 2024-04-15T15:23:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd995e47640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fd9900d1430 con 0x7fd9900cd000 2024-04-15T15:23:38.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9901602d0 0x7fd9900af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9901602d0 0x7fd9900af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53832/0 (socket says 172.21.15.12:53832) 2024-04-15T15:23:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 -- 172.21.15.12:0/3314758419 learned_addr learned my addr 172.21.15.12:0/3314758419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98f7fe640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd99009ab40 0x7fd99015ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd987fff640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9900cd000 0x7fd9900b7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 -- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9900cd000 msgr2=0x7fd9900b7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9900cd000 0x7fd9900b7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 -- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd99009ab40 msgr2=0x7fd99015ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd99009ab40 0x7fd99015ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 -- 172.21.15.12:0/3314758419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99015bf40 con 0x7fd9901602d0 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98ffff640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9901602d0 0x7fd9900af250 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd98800cef0 tx=0x7fd98800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98cff9640 1 -- 172.21.15.12:0/3314758419 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98801a070 con 0x7fd9901602d0 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.471+0000 7fd98cff9640 1 -- 172.21.15.12:0/3314758419 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98800d860 con 0x7fd9901602d0 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.472+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9901f8810 con 0x7fd9901602d0 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.472+0000 7fd98cff9640 1 -- 172.21.15.12:0/3314758419 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98800dc50 con 0x7fd9901602d0 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.472+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9901f8dd0 con 0x7fd9901602d0 2024-04-15T15:23:38.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.472+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9900689e0 con 0x7fd9901602d0 2024-04-15T15:23:38.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.473+0000 7fd98cff9640 1 -- 172.21.15.12:0/3314758419 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd98801d070 con 0x7fd9901602d0 2024-04-15T15:23:38.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.473+0000 7fd98cff9640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd97006c980 0x7fd97006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.473+0000 7fd98cff9640 1 -- 172.21.15.12:0/3314758419 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fd98808c210 con 0x7fd9901602d0 2024-04-15T15:23:38.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.474+0000 7fd98f7fe640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd97006c980 0x7fd97006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.474+0000 7fd98f7fe640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd97006c980 0x7fd97006ee40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd980007fd0 tx=0x7fd980007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.478+0000 7fd98cff9640 1 -- 172.21.15.12:0/3314758419 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd98805a270 con 0x7fd9901602d0 2024-04-15T15:23:38.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.659+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9900b4e20 con 0x7fd97006c980 2024-04-15T15:23:38.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.660+0000 7fd98cff9640 1 -- 172.21.15.12:0/3314758419 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13490 (secure 0 0 0) 0x7fd9900b4e20 con 0x7fd97006c980 2024-04-15T15:23:38.662 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:38.662 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:23:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd97006c980 msgr2=0x7fd97006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd97006c980 0x7fd97006ee40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd980007fd0 tx=0x7fd980007e80 comp rx=0 tx=0).stop 2024-04-15T15:23:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9901602d0 msgr2=0x7fd9900af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9901602d0 0x7fd9900af250 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd98800cef0 tx=0x7fd98800a710 comp rx=0 tx=0).stop 2024-04-15T15:23:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 shutdown_connections 2024-04-15T15:23:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9901602d0 0x7fd9900af250 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fd9900cd000 0x7fd9900b7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fd97006c980 0x7fd97006ee40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 --2- 172.21.15.12:0/3314758419 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd99009ab40 0x7fd99015ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.662+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 >> 172.21.15.12:0/3314758419 conn(0x7fd9900d2460 msgr2=0x7fd9900c4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.663+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 shutdown_connections 2024-04-15T15:23:38.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.663+0000 7fd995e47640 1 -- 172.21.15.12:0/3314758419 wait complete. 2024-04-15T15:23:38.675 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-04-15T15:23:38.624418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238164,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736556,"statfs":{"total":773094113280,"available":772850233344,"internally_reserved":0,"allocated":5447680,"data_stored":3047840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62581,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-130,"num_write":-96,"num_write_kb":-97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.903544"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":216,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-15T15:23:34.855100+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:23:34.855100+0000","last_peered":"2024-04-15T15:23:34.855100+0000","last_clean":"2024-04-15T15:23:34.855100+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:23:34.855100+0000","last_undegraded":"2024-04-15T15:23:34.855100+0000","last_fullsized":"2024-04-15T15:23:34.855100+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1126400,"data_stored":823756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27544,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1126400,"data_stored":823756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:23:38.675 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:23:38.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.781+0000 7f77e4288640 1 Processor -- start 2024-04-15T15:23:38.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.781+0000 7f77e4288640 1 -- start start 2024-04-15T15:23:38.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.782+0000 7f77e4288640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 0x7f77dc0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.782+0000 7f77e4288640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77dc0b47a0 con 0x7f77dc0d2360 2024-04-15T15:23:38.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.782+0000 7f77e4288640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77dc0b4a80 con 0x7f77dc15e700 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.782+0000 7f77e4288640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f77dc0b53b0 con 0x7f77dc0d3270 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.782+0000 7f77e1823640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f77dc0d2360 0x7f77dc0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50336/0 (socket says 172.21.15.12:50336) 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.782+0000 7f77e1823640 1 -- 172.21.15.12:0/2796670530 learned_addr learned my addr 172.21.15.12:0/2796670530 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097308616 0 0) 0x7f77dc0b47a0 con 0x7f77dc0d2360 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77c4003520 con 0x7f77dc0d2360 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e2825640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 0x7f77dc0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 952422615 0 0) 0x7f77c4003520 con 0x7f77dc0d2360 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 msgr2=0x7f77dc0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:38.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 0x7f77dc0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 >> v1:172.21.15.55:6789/0 conn(0x7f77dc15e700 legacy=0x7f77dc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77dc0b4d60 con 0x7f77dc0d2360 2024-04-15T15:23:38.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7f77d0001670 con 0x7f77dc0d2360 2024-04-15T15:23:38.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 >> v1:172.21.15.12:6789/0 conn(0x7f77dc0d2360 legacy=0x7f77dc0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 0x7f77c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77c4004d40 0x7f77c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f77dc0b53b0 con 0x7f77c40039e0 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f77c400baa0 con 0x7f77c4004d40 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f77c400bca0 con 0x7f77dc0d3270 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e2024640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1823640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 0x7f77c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e2825640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77c4004d40 0x7f77c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e2024640 1 -- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 msgr2=0x7f77c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e2024640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 0x7f77c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e2024640 1 -- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77c4004d40 msgr2=0x7f77c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:38.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e2024640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77c4004d40 0x7f77c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e2024640 1 -- 172.21.15.12:0/2796670530 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c4003520 con 0x7f77c40039e0 2024-04-15T15:23:38.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.783+0000 7f77e1823640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc0d3270 0x7f77c4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:38.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.784+0000 7f77e2024640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77c4003e00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f77d4007fd0 tx=0x7f77c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb602a239f95e62 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.784+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d40013c0 con 0x7f77c40039e0 2024-04-15T15:23:38.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.784+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77d40052d0 con 0x7f77c40039e0 2024-04-15T15:23:38.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.784+0000 7f77e1022640 1 -- 172.21.15.12:0/2796670530 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d4003640 con 0x7f77c40039e0 2024-04-15T15:23:38.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.785+0000 7f77e4288640 1 -- 172.21.15.12:0/2796670530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 msgr2=0x7f77c4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.785+0000 7f77e4288640 1 --2- 172.21.15.12:0/2796670530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77c4003e00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f77d4007fd0 tx=0x7f77c4005830 comp rx=0 tx=0).stop 2024-04-15T15:23:38.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.785+0000 7f77e2825640 1 -- 172.21.15.12:0/2796670530 reap_dead start 2024-04-15T15:23:38.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.785+0000 7f77e4288640 1 -- 172.21.15.12:0/2796670530 shutdown_connections 2024-04-15T15:23:38.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.785+0000 7f77e4288640 1 -- 172.21.15.12:0/2796670530 >> 172.21.15.12:0/2796670530 conn(0x7f77dc08bf60 msgr2=0x7f77dc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:38.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.785+0000 7f77e4288640 1 -- 172.21.15.12:0/2796670530 shutdown_connections 2024-04-15T15:23:38.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.785+0000 7f77e4288640 1 -- 172.21.15.12:0/2796670530 wait complete. 2024-04-15T15:23:38.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.786+0000 7f77e4288640 1 Processor -- start 2024-04-15T15:23:38.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.786+0000 7f77e4288640 1 -- start start 2024-04-15T15:23:38.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e4288640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77dc1f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e4288640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc15a9f0 0x7f77dc0cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e4288640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0ce330 0x7f77dc1f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e4288640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f77dc0d1d40 con 0x7f77c40039e0 2024-04-15T15:23:38.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e4288640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f77dc0b4450 con 0x7f77dc0ce330 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e4288640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f77dc0d1ec0 con 0x7f77dc15a9f0 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77dc1f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77dc1f5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53846/0 (socket says 172.21.15.12:53846) 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 -- 172.21.15.12:0/3641813001 learned_addr learned my addr 172.21.15.12:0/3641813001 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e1823640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc15a9f0 0x7f77dc0cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2825640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0ce330 0x7f77dc1f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 -- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc15a9f0 msgr2=0x7f77dc0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc15a9f0 0x7f77dc0cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 -- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0ce330 msgr2=0x7f77dc1f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0ce330 0x7f77dc1f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 -- 172.21.15.12:0/3641813001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77dc0cba30 con 0x7f77c40039e0 2024-04-15T15:23:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2825640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0ce330 0x7f77dc1f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.787+0000 7f77e2024640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77dc1f5170 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f77d402bc90 tx=0x7f77d4001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.788+0000 7f77daffd640 1 -- 172.21.15.12:0/3641813001 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d4041020 con 0x7f77c40039e0 2024-04-15T15:23:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.788+0000 7f77daffd640 1 -- 172.21.15.12:0/3641813001 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77d40050d0 con 0x7f77c40039e0 2024-04-15T15:23:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.788+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77dc0cbd10 con 0x7f77c40039e0 2024-04-15T15:23:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.788+0000 7f77daffd640 1 -- 172.21.15.12:0/3641813001 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d4007490 con 0x7f77c40039e0 2024-04-15T15:23:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.788+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77dc0cc2d0 con 0x7f77c40039e0 2024-04-15T15:23:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.789+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77dc0689e0 con 0x7f77c40039e0 2024-04-15T15:23:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.789+0000 7f77daffd640 1 -- 172.21.15.12:0/3641813001 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f77d4044020 con 0x7f77c40039e0 2024-04-15T15:23:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.790+0000 7f77daffd640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f77c806c980 0x7f77c806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.790+0000 7f77daffd640 1 -- 172.21.15.12:0/3641813001 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f77d40ad590 con 0x7f77c40039e0 2024-04-15T15:23:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.790+0000 7f77e1823640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f77c806c980 0x7f77c806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:38.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.791+0000 7f77e1823640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f77c806c980 0x7f77c806ee40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f77dc08e800 tx=0x7f77d0007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:38.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.794+0000 7f77daffd640 1 -- 172.21.15.12:0/3641813001 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77d404b030 con 0x7f77c40039e0 2024-04-15T15:23:38.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.981+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77dc0c81a0 con 0x7f77c40039e0 2024-04-15T15:23:38.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.982+0000 7f77daffd640 1 -- 172.21.15.12:0/3641813001 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+13387 (secure 0 0 0) 0x7f77d407ba70 con 0x7f77c40039e0 2024-04-15T15:23:38.983 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:38.984 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":45,"fsid":"928fa63b-2322-46eb-a6b8-ef2b880cd7c9","created":"2024-04-15T15:20:46.237113+0000","modified":"2024-04-15T15:23:34.713667+0000","last_up_change":"2024-04-15T15:20:57.511002+0000","last_in_change":"2024-04-15T15:20:48.528878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:20:58.603295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9af3971f-5436-411b-bd33-a0ec8913f792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6801","nonce":3293425032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6803","nonce":3293425032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6807","nonce":3293425032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3293425032},{"type":"v1","addr":"172.21.15.12:6805","nonce":3293425032}]},"public_addr":"172.21.15.12:6801/3293425032","cluster_addr":"172.21.15.12:6803/3293425032","heartbeat_back_addr":"172.21.15.12:6807/3293425032","heartbeat_front_addr":"172.21.15.12:6805/3293425032","state":["exists","up"]},{"osd":1,"uuid":"1235e7b5-5133-476f-b84d-315a337c9cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6817","nonce":1623484376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6819","nonce":1623484376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6823","nonce":1623484376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1623484376},{"type":"v1","addr":"172.21.15.12:6821","nonce":1623484376}]},"public_addr":"172.21.15.12:6817/1623484376","cluster_addr":"172.21.15.12:6819/1623484376","heartbeat_back_addr":"172.21.15.12:6823/1623484376","heartbeat_front_addr":"172.21.15.12:6821/1623484376","state":["exists","up"]},{"osd":2,"uuid":"b7f44bbb-0e6e-4acd-8218-14a0c31002bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6809","nonce":2859338982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6811","nonce":2859338982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6815","nonce":2859338982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2859338982},{"type":"v1","addr":"172.21.15.12:6813","nonce":2859338982}]},"public_addr":"172.21.15.12:6809/2859338982","cluster_addr":"172.21.15.12:6811/2859338982","heartbeat_back_addr":"172.21.15.12:6815/2859338982","heartbeat_front_addr":"172.21.15.12:6813/2859338982","state":["exists","up"]},{"osd":3,"uuid":"76b01b05-bf0f-4054-affd-821212dcd1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6825","nonce":3520688318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6827","nonce":3520688318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6831","nonce":3520688318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3520688318},{"type":"v1","addr":"172.21.15.12:6829","nonce":3520688318}]},"public_addr":"172.21.15.12:6825/3520688318","cluster_addr":"172.21.15.12:6827/3520688318","heartbeat_back_addr":"172.21.15.12:6831/3520688318","heartbeat_front_addr":"172.21.15.12:6829/3520688318","state":["exists","up"]},{"osd":4,"uuid":"824cae86-45ab-4f14-a657-0ac0235573aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6809","nonce":3874250091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6811","nonce":3874250091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6815","nonce":3874250091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3874250091},{"type":"v1","addr":"172.21.15.55:6813","nonce":3874250091}]},"public_addr":"172.21.15.55:6809/3874250091","cluster_addr":"172.21.15.55:6811/3874250091","heartbeat_back_addr":"172.21.15.55:6815/3874250091","heartbeat_front_addr":"172.21.15.55:6813/3874250091","state":["exists","up"]},{"osd":5,"uuid":"cc401892-8b26-4f18-aeff-38af9be45bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6825","nonce":4105274290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6827","nonce":4105274290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6831","nonce":4105274290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":4105274290},{"type":"v1","addr":"172.21.15.55:6829","nonce":4105274290}]},"public_addr":"172.21.15.55:6825/4105274290","cluster_addr":"172.21.15.55:6827/4105274290","heartbeat_back_addr":"172.21.15.55:6831/4105274290","heartbeat_front_addr":"172.21.15.55:6829/4105274290","state":["exists","up"]},{"osd":6,"uuid":"4a9b133b-ec62-4446-8e2c-3a1903c73fec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6817","nonce":2237244484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6819","nonce":2237244484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6823","nonce":2237244484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":2237244484},{"type":"v1","addr":"172.21.15.55:6821","nonce":2237244484}]},"public_addr":"172.21.15.55:6817/2237244484","cluster_addr":"172.21.15.55:6819/2237244484","heartbeat_back_addr":"172.21.15.55:6823/2237244484","heartbeat_front_addr":"172.21.15.55:6821/2237244484","state":["exists","up"]},{"osd":7,"uuid":"36106542-b6bf-439a-a645-5426a655af97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6801","nonce":4101047654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6803","nonce":4101047654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6807","nonce":4101047654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":4101047654},{"type":"v1","addr":"172.21.15.55:6805","nonce":4101047654}]},"public_addr":"172.21.15.55:6801/4101047654","cluster_addr":"172.21.15.55:6803/4101047654","heartbeat_back_addr":"172.21.15.55:6807/4101047654","heartbeat_front_addr":"172.21.15.55:6805/4101047654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.634943+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.721294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.592919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.707432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.740848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.716032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.766237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:20:53.708265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6834/1202155349":"2024-04-16T15:23:31.709260+0000","172.21.15.55:6836/1202155349":"2024-04-16T15:23:31.709260+0000","172.21.15.55:6837/3388147599":"2024-04-16T15:23:15.001839+0000","172.21.15.55:6835/3388147599":"2024-04-16T15:23:15.001839+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:23:38.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f77c806c980 msgr2=0x7f77c806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f77c806c980 0x7f77c806ee40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f77dc08e800 tx=0x7f77d0007040 comp rx=0 tx=0).stop 2024-04-15T15:23:38.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 msgr2=0x7f77dc1f5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:38.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77dc1f5170 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f77d402bc90 tx=0x7f77d4001170 comp rx=0 tx=0).stop 2024-04-15T15:23:38.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 shutdown_connections 2024-04-15T15:23:38.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f77c806c980 0x7f77c806ee40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0ce330 0x7f77dc1f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f77dc15a9f0 0x7f77dc0cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 --2- 172.21.15.12:0/3641813001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77c40039e0 0x7f77dc1f5170 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:38.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 >> 172.21.15.12:0/3641813001 conn(0x7f77dc08bf60 msgr2=0x7f77dc0cf330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:38.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 shutdown_connections 2024-04-15T15:23:38.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:38.984+0000 7f77e4288640 1 -- 172.21.15.12:0/3641813001 wait complete. 2024-04-15T15:23:39.998 INFO:tasks.ceph:Scrubbing osd.0 2024-04-15T15:23:39.998 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:23:40.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.119+0000 7f8a51550640 1 Processor -- start 2024-04-15T15:23:40.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.119+0000 7f8a51550640 1 -- start start 2024-04-15T15:23:40.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a51550640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8a4c15e8c0 0x7f8a4c0d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a51550640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0cfcb0 0x7f8a4c0d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a51550640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4c0d2340 con 0x7f8a4c0d0140 2024-04-15T15:23:40.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a51550640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8a4c15eca0 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a51550640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c0d2620 con 0x7f8a4c0cfcb0 2024-04-15T15:23:40.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a4affd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8a4c15e8c0 0x7f8a4c0d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a4b7fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f8a4c0d0140 0x7f8a4c06c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:43512/0 (socket says 172.21.15.12:43512) 2024-04-15T15:23:40.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a4b7fe640 1 -- 172.21.15.12:0/2601029315 learned_addr learned my addr 172.21.15.12:0/2601029315 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:40.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.120+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0cfcb0 0x7f8a4c0d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.121+0000 7f8a4affd640 1 -- 172.21.15.12:0/2601029315 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0cfcb0 msgr2=0x7f8a4c0d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:40.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.121+0000 7f8a4affd640 1 --2- 172.21.15.12:0/2601029315 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0cfcb0 0x7f8a4c0d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.121+0000 7f8a4affd640 1 -- 172.21.15.12:0/2601029315 >> v1:172.21.15.55:6789/0 conn(0x7f8a4c0d0140 legacy=0x7f8a4c06c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.121+0000 7f8a4a7fc640 1 -- 172.21.15.12:0/2601029315 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110910698 0 0) 0x7f8a4c0d2340 con 0x7f8a4c0d0140 2024-04-15T15:23:40.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.121+0000 7f8a4affd640 1 -- 172.21.15.12:0/2601029315 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0d2900 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.122+0000 7f8a4affd640 1 --2- 172.21.15.12:0/2601029315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a4c15e8c0 0x7f8a4c0d3100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0097a0 tx=0x7f8a3c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3e202efc6d8c125 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.122+0000 7f8a4a7fc640 1 -- 172.21.15.12:0/2601029315 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c035020 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.122+0000 7f8a4a7fc640 1 -- 172.21.15.12:0/2601029315 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3c031830 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.122+0000 7f8a4a7fc640 1 -- 172.21.15.12:0/2601029315 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c031c00 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.122+0000 7f8a51550640 1 -- 172.21.15.12:0/2601029315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a4c15e8c0 msgr2=0x7f8a4c0d3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.122+0000 7f8a51550640 1 --2- 172.21.15.12:0/2601029315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a4c15e8c0 0x7f8a4c0d3100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0097a0 tx=0x7f8a3c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.123+0000 7f8a51550640 1 -- 172.21.15.12:0/2601029315 shutdown_connections 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.123+0000 7f8a51550640 1 --2- 172.21.15.12:0/2601029315 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0cfcb0 0x7f8a4c0d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.123+0000 7f8a51550640 1 --2- 172.21.15.12:0/2601029315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a4c15e8c0 0x7f8a4c0d3100 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.123+0000 7f8a51550640 1 -- 172.21.15.12:0/2601029315 >> 172.21.15.12:0/2601029315 conn(0x7f8a4c08bf60 msgr2=0x7f8a4c0d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:40.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.123+0000 7f8a51550640 1 -- 172.21.15.12:0/2601029315 shutdown_connections 2024-04-15T15:23:40.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.123+0000 7f8a51550640 1 -- 172.21.15.12:0/2601029315 wait complete. 2024-04-15T15:23:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.124+0000 7f8a51550640 1 Processor -- start 2024-04-15T15:23:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.124+0000 7f8a51550640 1 -- start start 2024-04-15T15:23:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.124+0000 7f8a51550640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a4c0cfcb0 0x7f8a4c0b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.124+0000 7f8a51550640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0d0140 0x7f8a4c14e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.124+0000 7f8a51550640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a4c15e8c0 0x7f8a4c160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a51550640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c0d4f60 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a51550640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c0d4de0 con 0x7f8a4c0cfcb0 2024-04-15T15:23:40.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a51550640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c0d13c0 con 0x7f8a4c0d0140 2024-04-15T15:23:40.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a4c15e8c0 0x7f8a4c160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a4c15e8c0 0x7f8a4c160ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53862/0 (socket says 172.21.15.12:53862) 2024-04-15T15:23:40.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 -- 172.21.15.12:0/2861491057 learned_addr learned my addr 172.21.15.12:0/2861491057 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:40.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4affd640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a4c0cfcb0 0x7f8a4c0b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a43fff640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0d0140 0x7f8a4c14e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 -- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0d0140 msgr2=0x7f8a4c14e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f8a4c0d0140 0x7f8a4c14e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 -- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a4c0cfcb0 msgr2=0x7f8a4c0b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a4c0cfcb0 0x7f8a4c0b71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 -- 172.21.15.12:0/2861491057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0b7710 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a4b7fe640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a4c15e8c0 0x7f8a4c160ff0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400c7d0 tx=0x7f8a4400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44018070 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a4400a640 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4400aa70 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.125+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c0b79b0 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.126+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c0b7f70 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.126+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8a4c0689e0 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.127+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8a4401b070 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.127+0000 7f8a437fe640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8a2c06c980 0x7f8a2c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.128+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f8a4408afb0 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.128+0000 7f8a437fe640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7f8a2c072ec0 0x7f8a2c075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.128+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 --> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a4401be50 con 0x7f8a2c072ec0 2024-04-15T15:23:40.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.128+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f8a44092050 con 0x7f8a4c15e8c0 2024-04-15T15:23:40.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.128+0000 7f8a4affd640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8a2c06c980 0x7f8a2c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.128+0000 7f8a43fff640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7f8a2c072ec0 0x7f8a2c075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.128+0000 7f8a43fff640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7f8a2c072ec0 0x7f8a2c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.128+0000 7f8a4affd640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8a2c06c980 0x7f8a2c06ee40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c036000 tx=0x7f8a3c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.129+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 <== osd.0 v2:172.21.15.12:6800/3293425032 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a4401be50 con 0x7f8a2c072ec0 2024-04-15T15:23:40.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.142+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 --> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8a4c0d1790 con 0x7f8a2c072ec0 2024-04-15T15:23:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.142+0000 7f8a437fe640 1 -- 172.21.15.12:0/2861491057 <== osd.0 v2:172.21.15.12:6800/3293425032 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8a4c0d1790 con 0x7f8a2c072ec0 2024-04-15T15:23:40.144 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:40.144 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:23:40.144 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7f8a2c072ec0 msgr2=0x7f8a2c075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.12:6800/3293425032,v1:172.21.15.12:6801/3293425032] conn(0x7f8a2c072ec0 0x7f8a2c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8a2c06c980 msgr2=0x7f8a2c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f8a2c06c980 0x7f8a2c06ee40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c036000 tx=0x7f8a3c008030 comp rx=0 tx=0).stop 2024-04-15T15:23:40.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a4c15e8c0 msgr2=0x7f8a4c160ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 --2- 172.21.15.12:0/2861491057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a4c15e8c0 0x7f8a4c160ff0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400c7d0 tx=0x7f8a4400cc90 comp rx=0 tx=0).stop 2024-04-15T15:23:40.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a4b7fe640 1 -- 172.21.15.12:0/2861491057 reap_dead start 2024-04-15T15:23:40.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 shutdown_connections 2024-04-15T15:23:40.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 >> 172.21.15.12:0/2861491057 conn(0x7f8a4c08bf60 msgr2=0x7f8a4c0b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:40.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 shutdown_connections 2024-04-15T15:23:40.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.143+0000 7f8a51550640 1 -- 172.21.15.12:0/2861491057 wait complete. 2024-04-15T15:23:40.155 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-15T15:23:40.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.261+0000 7fac3b18d640 1 Processor -- start 2024-04-15T15:23:40.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.261+0000 7fac3b18d640 1 -- start start 2024-04-15T15:23:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.261+0000 7fac3b18d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac3409ac40 0x7fac340d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.261+0000 7fac3b18d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340b49d0 0x7fac340ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.261+0000 7fac3b18d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac340d1210 con 0x7fac340ce570 2024-04-15T15:23:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.261+0000 7fac3b18d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fac340b1b70 con 0x7fac3409ac40 2024-04-15T15:23:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.261+0000 7fac3b18d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fac340d14b0 con 0x7fac340b49d0 2024-04-15T15:23:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac38f29640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac3409ac40 0x7fac340d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac38f29640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fac3409ac40 0x7fac340d0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53874/0 (socket says 172.21.15.12:53874) 2024-04-15T15:23:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac38f29640 1 -- 172.21.15.12:0/2501437438 learned_addr learned my addr 172.21.15.12:0/2501437438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac33fff640 1 --2- 172.21.15.12:0/2501437438 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340b49d0 0x7fac340ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac38f29640 1 -- 172.21.15.12:0/2501437438 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340b49d0 msgr2=0x7fac340ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac38f29640 1 --2- 172.21.15.12:0/2501437438 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340b49d0 0x7fac340ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac38f29640 1 -- 172.21.15.12:0/2501437438 >> v1:172.21.15.12:6789/0 conn(0x7fac340ce570 legacy=0x7fac340b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac38f29640 1 -- 172.21.15.12:0/2501437438 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac340cd550 con 0x7fac3409ac40 2024-04-15T15:23:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.262+0000 7fac33fff640 1 --2- 172.21.15.12:0/2501437438 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340b49d0 0x7fac340ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.263+0000 7fac38f29640 1 --2- 172.21.15.12:0/2501437438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac340d0cd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fac28009870 tx=0x7fac2802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a41c83460776b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.263+0000 7fac337fe640 1 -- 172.21.15.12:0/2501437438 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2802f840 con 0x7fac3409ac40 2024-04-15T15:23:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.263+0000 7fac337fe640 1 -- 172.21.15.12:0/2501437438 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2802f9e0 con 0x7fac3409ac40 2024-04-15T15:23:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.263+0000 7fac337fe640 1 -- 172.21.15.12:0/2501437438 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2802fdb0 con 0x7fac3409ac40 2024-04-15T15:23:40.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.264+0000 7fac3b18d640 1 -- 172.21.15.12:0/2501437438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 msgr2=0x7fac340d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.264+0000 7fac3b18d640 1 --2- 172.21.15.12:0/2501437438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac340d0cd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fac28009870 tx=0x7fac2802ba20 comp rx=0 tx=0).stop 2024-04-15T15:23:40.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.264+0000 7fac3b18d640 1 -- 172.21.15.12:0/2501437438 shutdown_connections 2024-04-15T15:23:40.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.264+0000 7fac3b18d640 1 --2- 172.21.15.12:0/2501437438 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340b49d0 0x7fac340ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.264+0000 7fac3b18d640 1 --2- 172.21.15.12:0/2501437438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac340d0cd0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.264+0000 7fac3b18d640 1 -- 172.21.15.12:0/2501437438 >> 172.21.15.12:0/2501437438 conn(0x7fac340b3eb0 msgr2=0x7fac340b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.264+0000 7fac3b18d640 1 -- 172.21.15.12:0/2501437438 shutdown_connections 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.264+0000 7fac3b18d640 1 -- 172.21.15.12:0/2501437438 wait complete. 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.265+0000 7fac3b18d640 1 Processor -- start 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.265+0000 7fac3b18d640 1 -- start start 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac3b18d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac3406e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac3b18d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac340b49d0 0x7fac341e60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac3b18d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340ce570 0x7fac341e6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac3b18d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fac340b2430 con 0x7fac3409ac40 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac3b18d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fac340cd1e0 con 0x7fac340b49d0 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac3b18d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fac340b25b0 con 0x7fac340ce570 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac3406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac3406e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53886/0 (socket says 172.21.15.12:53886) 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 -- 172.21.15.12:0/4249977330 learned_addr learned my addr 172.21.15.12:0/4249977330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac33fff640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac340b49d0 0x7fac341e60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac3972a640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340ce570 0x7fac341e6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 -- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340ce570 msgr2=0x7fac341e6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340ce570 0x7fac341e6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 -- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac340b49d0 msgr2=0x7fac341e60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac340b49d0 0x7fac341e60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 -- 172.21.15.12:0/4249977330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac340b6d10 con 0x7fac3409ac40 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac33fff640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac340b49d0 0x7fac341e60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac3972a640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340ce570 0x7fac341e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.266+0000 7fac38f29640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac3406e230 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fac28009870 tx=0x7fac28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.267+0000 7fac317fa640 1 -- 172.21.15.12:0/4249977330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28041400 con 0x7fac3409ac40 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.267+0000 7fac317fa640 1 -- 172.21.15.12:0/4249977330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac28046070 con 0x7fac3409ac40 2024-04-15T15:23:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.267+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac340b6ff0 con 0x7fac3409ac40 2024-04-15T15:23:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.267+0000 7fac317fa640 1 -- 172.21.15.12:0/4249977330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28041800 con 0x7fac3409ac40 2024-04-15T15:23:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.267+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac340b75b0 con 0x7fac3409ac40 2024-04-15T15:23:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.268+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac340689e0 con 0x7fac3409ac40 2024-04-15T15:23:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.268+0000 7fac317fa640 1 -- 172.21.15.12:0/4249977330 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fac28040070 con 0x7fac3409ac40 2024-04-15T15:23:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.269+0000 7fac317fa640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac1806c980 0x7fac1806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.269+0000 7fac317fa640 1 -- 172.21.15.12:0/4249977330 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fac280ae470 con 0x7fac3409ac40 2024-04-15T15:23:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.269+0000 7fac33fff640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac1806c980 0x7fac1806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.269+0000 7fac33fff640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac1806c980 0x7fac1806ee40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fac20006fd0 tx=0x7fac20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.273+0000 7fac317fa640 1 -- 172.21.15.12:0/4249977330 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac28033030 con 0x7fac3409ac40 2024-04-15T15:23:40.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.462+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fac3409b500 con 0x7fac1806c980 2024-04-15T15:23:40.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.463+0000 7fac317fa640 1 -- 172.21.15.12:0/4249977330 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fac3409b500 con 0x7fac1806c980 2024-04-15T15:23:40.464 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 0 to deep-scrub 2024-04-15T15:23:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac1806c980 msgr2=0x7fac1806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac1806c980 0x7fac1806ee40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fac20006fd0 tx=0x7fac20008040 comp rx=0 tx=0).stop 2024-04-15T15:23:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 msgr2=0x7fac3406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac3406e230 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fac28009870 tx=0x7fac28004730 comp rx=0 tx=0).stop 2024-04-15T15:23:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 shutdown_connections 2024-04-15T15:23:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fac340ce570 0x7fac341e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fac1806c980 0x7fac1806ee40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac340b49d0 0x7fac341e60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 --2- 172.21.15.12:0/4249977330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac3409ac40 0x7fac3406e230 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 >> 172.21.15.12:0/4249977330 conn(0x7fac340b3eb0 msgr2=0x7fac34070260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:40.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 shutdown_connections 2024-04-15T15:23:40.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.465+0000 7fac3b18d640 1 -- 172.21.15.12:0/4249977330 wait complete. 2024-04-15T15:23:40.477 INFO:tasks.ceph:Scrubbing osd.1 2024-04-15T15:23:40.478 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:23:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.585+0000 7fdf2f53d640 1 Processor -- start 2024-04-15T15:23:40.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.586+0000 7fdf2f53d640 1 -- start start 2024-04-15T15:23:40.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.586+0000 7fdf2f53d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 0x7fdf2806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.586+0000 7fdf2f53d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf280cbdb0 con 0x7fdf280cb910 2024-04-15T15:23:40.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.586+0000 7fdf2f53d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf280cc090 con 0x7fdf280d0470 2024-04-15T15:23:40.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.586+0000 7fdf2f53d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fdf280b4e80 con 0x7fdf2815c430 2024-04-15T15:23:40.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf2cad8640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdf280cb910 0x7fdf280b4130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50352/0 (socket says 172.21.15.12:50352) 2024-04-15T15:23:40.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf2cad8640 1 -- 172.21.15.12:0/2828315915 learned_addr learned my addr 172.21.15.12:0/2828315915 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:40.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf2d2d9640 1 --2- 172.21.15.12:0/2828315915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 0x7fdf2806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf1bfff640 1 -- 172.21.15.12:0/2828315915 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1492436270 0 0) 0x7fdf280cbdb0 con 0x7fdf280cb910 2024-04-15T15:23:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf1bfff640 1 -- 172.21.15.12:0/2828315915 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf14003520 con 0x7fdf280cb910 2024-04-15T15:23:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf1bfff640 1 -- 172.21.15.12:0/2828315915 <== mon.1 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147808595 0 0) 0x7fdf280cc090 con 0x7fdf280d0470 2024-04-15T15:23:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf1bfff640 1 -- 172.21.15.12:0/2828315915 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf280cbdb0 con 0x7fdf280d0470 2024-04-15T15:23:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf2d2d9640 1 -- 172.21.15.12:0/2828315915 >> v1:172.21.15.55:6789/0 conn(0x7fdf280d0470 legacy=0x7fdf280cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf2d2d9640 1 -- 172.21.15.12:0/2828315915 >> v1:172.21.15.12:6789/0 conn(0x7fdf280cb910 legacy=0x7fdf280b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.587+0000 7fdf2d2d9640 1 -- 172.21.15.12:0/2828315915 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf280cc370 con 0x7fdf2815c430 2024-04-15T15:23:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.588+0000 7fdf2d2d9640 1 --2- 172.21.15.12:0/2828315915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 0x7fdf2806dce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf240097d0 tx=0x7fdf2402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85b9546e7dc94827 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.588+0000 7fdf1bfff640 1 -- 172.21.15.12:0/2828315915 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf24035020 con 0x7fdf2815c430 2024-04-15T15:23:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.588+0000 7fdf1bfff640 1 -- 172.21.15.12:0/2828315915 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf24031800 con 0x7fdf2815c430 2024-04-15T15:23:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.588+0000 7fdf1bfff640 1 -- 172.21.15.12:0/2828315915 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf24031bb0 con 0x7fdf2815c430 2024-04-15T15:23:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.589+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2828315915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 msgr2=0x7fdf2806dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.589+0000 7fdf2f53d640 1 --2- 172.21.15.12:0/2828315915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 0x7fdf2806dce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf240097d0 tx=0x7fdf2402b8a0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.589+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2828315915 shutdown_connections 2024-04-15T15:23:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.589+0000 7fdf2f53d640 1 --2- 172.21.15.12:0/2828315915 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 0x7fdf2806dce0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.589+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2828315915 >> 172.21.15.12:0/2828315915 conn(0x7fdf2806bbc0 msgr2=0x7fdf2806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.589+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2828315915 shutdown_connections 2024-04-15T15:23:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.589+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2828315915 wait complete. 2024-04-15T15:23:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.590+0000 7fdf2f53d640 1 Processor -- start 2024-04-15T15:23:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.590+0000 7fdf2f53d640 1 -- start start 2024-04-15T15:23:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2f53d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf280cb910 0x7fdf2815cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2f53d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280d0470 0x7fdf28155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2f53d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 0x7fdf2815d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2f53d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2815c810 con 0x7fdf280d0470 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2f53d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdf280cc930 con 0x7fdf280cb910 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2f53d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fdf2815c990 con 0x7fdf2815c430 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280d0470 0x7fdf28155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280d0470 0x7fdf28155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53890/0 (socket says 172.21.15.12:53890) 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 -- 172.21.15.12:0/2935219730 learned_addr learned my addr 172.21.15.12:0/2935219730 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2d2d9640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf280cb910 0x7fdf2815cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2dada640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 0x7fdf2815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 -- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 msgr2=0x7fdf2815d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fdf2815c430 0x7fdf2815d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 -- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf280cb910 msgr2=0x7fdf2815cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf280cb910 0x7fdf2815cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 -- 172.21.15.12:0/2935219730 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf280c7300 con 0x7fdf280d0470 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.591+0000 7fdf2cad8640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280d0470 0x7fdf28155860 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00c500 tx=0x7fdf1c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.592+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c017070 con 0x7fdf280d0470 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.592+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf28099df0 con 0x7fdf280d0470 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.592+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf1c005ce0 con 0x7fdf280d0470 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.592+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c0072f0 con 0x7fdf280d0470 2024-04-15T15:23:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.592+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf28099b10 con 0x7fdf280d0470 2024-04-15T15:23:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.593+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdf280689e0 con 0x7fdf280d0470 2024-04-15T15:23:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.593+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fdf1c01a070 con 0x7fdf280d0470 2024-04-15T15:23:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf19ffb640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf0c06c980 0x7fdf0c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fdf1c08ae70 con 0x7fdf280d0470 2024-04-15T15:23:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf19ffb640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7fdf0c072ec0 0x7fdf0c075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 --> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf1c013e50 con 0x7fdf0c072ec0 2024-04-15T15:23:40.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fdf1c08d5c0 con 0x7fdf280d0470 2024-04-15T15:23:40.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf2d2d9640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf0c06c980 0x7fdf0c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf2dada640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7fdf0c072ec0 0x7fdf0c075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf2dada640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7fdf0c072ec0 0x7fdf0c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.594+0000 7fdf2d2d9640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf0c06c980 0x7fdf0c06ee40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdf24036000 tx=0x7fdf2403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.595+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 <== osd.1 v2:172.21.15.12:6816/1623484376 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdf1c013e50 con 0x7fdf0c072ec0 2024-04-15T15:23:40.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 --> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdf2815b9a0 con 0x7fdf0c072ec0 2024-04-15T15:23:40.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf19ffb640 1 -- 172.21.15.12:0/2935219730 <== osd.1 v2:172.21.15.12:6816/1623484376 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdf2815b9a0 con 0x7fdf0c072ec0 2024-04-15T15:23:40.625 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:40.625 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:23:40.626 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:40.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7fdf0c072ec0 msgr2=0x7fdf0c075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.12:6816/1623484376,v1:172.21.15.12:6817/1623484376] conn(0x7fdf0c072ec0 0x7fdf0c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf0c06c980 msgr2=0x7fdf0c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fdf0c06c980 0x7fdf0c06ee40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdf24036000 tx=0x7fdf2403c040 comp rx=0 tx=0).stop 2024-04-15T15:23:40.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280d0470 msgr2=0x7fdf28155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 --2- 172.21.15.12:0/2935219730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280d0470 0x7fdf28155860 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00c500 tx=0x7fdf1c00c9c0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2dada640 1 -- 172.21.15.12:0/2935219730 reap_dead start 2024-04-15T15:23:40.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 shutdown_connections 2024-04-15T15:23:40.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 >> 172.21.15.12:0/2935219730 conn(0x7fdf2806bbc0 msgr2=0x7fdf2806ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:40.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 shutdown_connections 2024-04-15T15:23:40.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.624+0000 7fdf2f53d640 1 -- 172.21.15.12:0/2935219730 wait complete. 2024-04-15T15:23:40.636 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-15T15:23:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.743+0000 7fda5bd8d640 1 Processor -- start 2024-04-15T15:23:40.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.744+0000 7fda5bd8d640 1 -- start start 2024-04-15T15:23:40.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.744+0000 7fda5bd8d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fda540cf600 0x7fda540cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.744+0000 7fda5bd8d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda540af4e0 0x7fda540af950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.744+0000 7fda5bd8d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda540ae440 con 0x7fda5415ed00 2024-04-15T15:23:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.744+0000 7fda5bd8d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fda54071030 con 0x7fda540cf600 2024-04-15T15:23:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.744+0000 7fda5bd8d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fda540ae720 con 0x7fda540af4e0 2024-04-15T15:23:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.744+0000 7fda59b29640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fda5415ed00 0x7fda54070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50356/0 (socket says 172.21.15.12:50356) 2024-04-15T15:23:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda59b29640 1 -- 172.21.15.12:0/3427321200 learned_addr learned my addr 172.21.15.12:0/3427321200 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda59328640 1 --2- 172.21.15.12:0/3427321200 >> v2:172.21.15.55:3300/0 conn(0x7fda540cf600 0x7fda540cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda5a32a640 1 --2- 172.21.15.12:0/3427321200 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda540af4e0 0x7fda540af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda58b27640 1 -- 172.21.15.12:0/3427321200 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242969768 0 0) 0x7fda540ae440 con 0x7fda5415ed00 2024-04-15T15:23:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda58b27640 1 -- 172.21.15.12:0/3427321200 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda3c003520 con 0x7fda5415ed00 2024-04-15T15:23:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda59328640 1 -- 172.21.15.12:0/3427321200 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda540af4e0 msgr2=0x7fda540af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda59328640 1 --2- 172.21.15.12:0/3427321200 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda540af4e0 0x7fda540af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda59328640 1 -- 172.21.15.12:0/3427321200 >> v1:172.21.15.12:6789/0 conn(0x7fda5415ed00 legacy=0x7fda54070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.745+0000 7fda59328640 1 -- 172.21.15.12:0/3427321200 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda540aea00 con 0x7fda540cf600 2024-04-15T15:23:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.746+0000 7fda59328640 1 --2- 172.21.15.12:0/3427321200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540cf600 0x7fda540cfa90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fda48009d90 tx=0x7fda4802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2e7aca9445670cb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.751+0000 7fda58b27640 1 -- 172.21.15.12:0/3427321200 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48039070 con 0x7fda540cf600 2024-04-15T15:23:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.751+0000 7fda58b27640 1 -- 172.21.15.12:0/3427321200 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda4802cc50 con 0x7fda540cf600 2024-04-15T15:23:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.751+0000 7fda58b27640 1 -- 172.21.15.12:0/3427321200 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48034440 con 0x7fda540cf600 2024-04-15T15:23:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.752+0000 7fda5bd8d640 1 -- 172.21.15.12:0/3427321200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540cf600 msgr2=0x7fda540cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.752+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/3427321200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540cf600 0x7fda540cfa90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fda48009d90 tx=0x7fda4802bea0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.752+0000 7fda5bd8d640 1 -- 172.21.15.12:0/3427321200 shutdown_connections 2024-04-15T15:23:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.752+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/3427321200 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda540af4e0 0x7fda540af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.752+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/3427321200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540cf600 0x7fda540cfa90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.752+0000 7fda5bd8d640 1 -- 172.21.15.12:0/3427321200 >> 172.21.15.12:0/3427321200 conn(0x7fda5409a9d0 msgr2=0x7fda540d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.752+0000 7fda5bd8d640 1 -- 172.21.15.12:0/3427321200 shutdown_connections 2024-04-15T15:23:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.752+0000 7fda5bd8d640 1 -- 172.21.15.12:0/3427321200 wait complete. 2024-04-15T15:23:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.753+0000 7fda5bd8d640 1 Processor -- start 2024-04-15T15:23:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.753+0000 7fda5bd8d640 1 -- start start 2024-04-15T15:23:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda5bd8d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540af4e0 0x7fda54157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda5bd8d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda540cf600 0x7fda540cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda5bd8d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda5415ed00 0x7fda5415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda5bd8d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda540701a0 con 0x7fda540cf600 2024-04-15T15:23:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda5bd8d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fda540aefc0 con 0x7fda540af4e0 2024-04-15T15:23:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda5bd8d640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fda540afe90 con 0x7fda5415ed00 2024-04-15T15:23:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59328640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda540cf600 0x7fda540cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59328640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda540cf600 0x7fda540cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53896/0 (socket says 172.21.15.12:53896) 2024-04-15T15:23:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59328640 1 -- 172.21.15.12:0/601745047 learned_addr learned my addr 172.21.15.12:0/601745047 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda5a32a640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda5415ed00 0x7fda5415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59b29640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540af4e0 0x7fda54157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59328640 1 -- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda5415ed00 msgr2=0x7fda5415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59328640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda5415ed00 0x7fda5415d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59328640 1 -- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540af4e0 msgr2=0x7fda54157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59328640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540af4e0 0x7fda54157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59328640 1 -- 172.21.15.12:0/601745047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda540cbf90 con 0x7fda540cf600 2024-04-15T15:23:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.754+0000 7fda59b29640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540af4e0 0x7fda54157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.755+0000 7fda59328640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda540cf600 0x7fda540cce40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fda48009d90 tx=0x7fda48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.755+0000 7fda527fc640 1 -- 172.21.15.12:0/601745047 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48044020 con 0x7fda540cf600 2024-04-15T15:23:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.755+0000 7fda527fc640 1 -- 172.21.15.12:0/601745047 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda4802ca50 con 0x7fda540cf600 2024-04-15T15:23:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.755+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda540cd870 con 0x7fda540cf600 2024-04-15T15:23:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.755+0000 7fda527fc640 1 -- 172.21.15.12:0/601745047 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48039040 con 0x7fda540cf600 2024-04-15T15:23:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.755+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda540cc510 con 0x7fda540cf600 2024-04-15T15:23:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.756+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda540690f0 con 0x7fda540cf600 2024-04-15T15:23:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.756+0000 7fda527fc640 1 -- 172.21.15.12:0/601745047 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fda48047020 con 0x7fda540cf600 2024-04-15T15:23:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.757+0000 7fda527fc640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda4006c980 0x7fda4006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.757+0000 7fda527fc640 1 -- 172.21.15.12:0/601745047 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fda480ae240 con 0x7fda540cf600 2024-04-15T15:23:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.757+0000 7fda59b29640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda4006c980 0x7fda4006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.758+0000 7fda59b29640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda4006c980 0x7fda4006ee40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fda44007fd0 tx=0x7fda44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.761+0000 7fda527fc640 1 -- 172.21.15.12:0/601745047 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda4804e030 con 0x7fda540cf600 2024-04-15T15:23:40.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.947+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fda5415ccc0 con 0x7fda4006c980 2024-04-15T15:23:40.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.947+0000 7fda527fc640 1 -- 172.21.15.12:0/601745047 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fda5415ccc0 con 0x7fda4006c980 2024-04-15T15:23:40.949 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 1 to deep-scrub 2024-04-15T15:23:40.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda4006c980 msgr2=0x7fda4006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda4006c980 0x7fda4006ee40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fda44007fd0 tx=0x7fda44009040 comp rx=0 tx=0).stop 2024-04-15T15:23:40.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda540cf600 msgr2=0x7fda540cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:40.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda540cf600 0x7fda540cce40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fda48009d90 tx=0x7fda48004a40 comp rx=0 tx=0).stop 2024-04-15T15:23:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 shutdown_connections 2024-04-15T15:23:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fda5415ed00 0x7fda5415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda540cf600 0x7fda540cce40 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fda4006c980 0x7fda4006ee40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 --2- 172.21.15.12:0/601745047 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda540af4e0 0x7fda54157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 >> 172.21.15.12:0/601745047 conn(0x7fda5409a9d0 msgr2=0x7fda540b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 shutdown_connections 2024-04-15T15:23:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:40.949+0000 7fda5bd8d640 1 -- 172.21.15.12:0/601745047 wait complete. 2024-04-15T15:23:40.961 INFO:tasks.ceph:Scrubbing osd.2 2024-04-15T15:23:40.962 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:23:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.070+0000 7f05b2f75640 1 Processor -- start 2024-04-15T15:23:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.070+0000 7f05b2f75640 1 -- start start 2024-04-15T15:23:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.070+0000 7f05b2f75640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f05ac15e8c0 0x7f05ac0d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.070+0000 7f05b2f75640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac0cfcb0 0x7f05ac0d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.070+0000 7f05b2f75640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05ac0d2340 con 0x7f05ac0d0140 2024-04-15T15:23:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05b2f75640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f05ac15eca0 con 0x7f05ac15e8c0 2024-04-15T15:23:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05b2f75640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f05ac0d2620 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05b1512640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f05ac0d0140 0x7f05ac06c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50358/0 (socket says 172.21.15.12:50358) 2024-04-15T15:23:41.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05b1512640 1 -- 172.21.15.12:0/471132574 learned_addr learned my addr 172.21.15.12:0/471132574 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05abfff640 1 --2- 172.21.15.12:0/471132574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac0cfcb0 0x7f05ac0d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05b0d11640 1 --2- 172.21.15.12:0/471132574 >> v2:172.21.15.55:3300/0 conn(0x7f05ac15e8c0 0x7f05ac0d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05ab7fe640 1 -- 172.21.15.12:0/471132574 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818131076 0 0) 0x7f05ac0d2340 con 0x7f05ac0d0140 2024-04-15T15:23:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05ab7fe640 1 -- 172.21.15.12:0/471132574 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f059c003520 con 0x7f05ac0d0140 2024-04-15T15:23:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05ab7fe640 1 -- 172.21.15.12:0/471132574 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2529819704 0 0) 0x7f059c003520 con 0x7f05ac0d0140 2024-04-15T15:23:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05abfff640 1 -- 172.21.15.12:0/471132574 >> v2:172.21.15.55:3300/0 conn(0x7f05ac15e8c0 msgr2=0x7f05ac0d3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05abfff640 1 --2- 172.21.15.12:0/471132574 >> v2:172.21.15.55:3300/0 conn(0x7f05ac15e8c0 0x7f05ac0d3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05abfff640 1 -- 172.21.15.12:0/471132574 >> v1:172.21.15.12:6789/0 conn(0x7f05ac0d0140 legacy=0x7f05ac06c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05abfff640 1 -- 172.21.15.12:0/471132574 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ac0d2900 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.071+0000 7f05b0d11640 1 --2- 172.21.15.12:0/471132574 >> v2:172.21.15.55:3300/0 conn(0x7f05ac15e8c0 0x7f05ac0d3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:41.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.072+0000 7f05abfff640 1 --2- 172.21.15.12:0/471132574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac0cfcb0 0x7f05ac0d0e80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f05980097a0 tx=0x7f059802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed950b3ff1fce1a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.072+0000 7f05ab7fe640 1 -- 172.21.15.12:0/471132574 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598037070 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.072+0000 7f05ab7fe640 1 -- 172.21.15.12:0/471132574 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0598030510 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.072+0000 7f05ab7fe640 1 -- 172.21.15.12:0/471132574 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a40019f0 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.073+0000 7f05b2f75640 1 -- 172.21.15.12:0/471132574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac0cfcb0 msgr2=0x7f05ac0d0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.073+0000 7f05b2f75640 1 --2- 172.21.15.12:0/471132574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac0cfcb0 0x7f05ac0d0e80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f05980097a0 tx=0x7f059802b790 comp rx=0 tx=0).stop 2024-04-15T15:23:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.073+0000 7f05b2f75640 1 -- 172.21.15.12:0/471132574 shutdown_connections 2024-04-15T15:23:41.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.073+0000 7f05b2f75640 1 --2- 172.21.15.12:0/471132574 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac0cfcb0 0x7f05ac0d0e80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.073+0000 7f05b2f75640 1 --2- 172.21.15.12:0/471132574 >> v2:172.21.15.55:3300/0 conn(0x7f05ac15e8c0 0x7f05ac0d3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.073+0000 7f05b2f75640 1 -- 172.21.15.12:0/471132574 >> 172.21.15.12:0/471132574 conn(0x7f05ac08bf60 msgr2=0x7f05ac0d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.073+0000 7f05b2f75640 1 -- 172.21.15.12:0/471132574 shutdown_connections 2024-04-15T15:23:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.073+0000 7f05b2f75640 1 -- 172.21.15.12:0/471132574 wait complete. 2024-04-15T15:23:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.074+0000 7f05b2f75640 1 Processor -- start 2024-04-15T15:23:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.074+0000 7f05b2f75640 1 -- start start 2024-04-15T15:23:41.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b2f75640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac0cfcb0 0x7f05ac0b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b2f75640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0d0140 0x7f05ac0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b2f75640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac15e8c0 0x7f05ac15cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b2f75640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac0d4f60 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b2f75640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac0d4de0 con 0x7f05ac0d0140 2024-04-15T15:23:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b2f75640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f05ac0d13c0 con 0x7f05ac15e8c0 2024-04-15T15:23:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b0d11640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac0cfcb0 0x7f05ac0b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b0d11640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac0cfcb0 0x7f05ac0b8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53910/0 (socket says 172.21.15.12:53910) 2024-04-15T15:23:41.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b0d11640 1 -- 172.21.15.12:0/1183212269 learned_addr learned my addr 172.21.15.12:0/1183212269 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b1512640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac15e8c0 0x7f05ac15cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05abfff640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0d0140 0x7f05ac0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b0d11640 1 -- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac15e8c0 msgr2=0x7f05ac15cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b0d11640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac15e8c0 0x7f05ac15cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.075+0000 7f05b0d11640 1 -- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0d0140 msgr2=0x7f05ac0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05b0d11640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0d0140 0x7f05ac0b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05b0d11640 1 -- 172.21.15.12:0/1183212269 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ac0b9460 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05b1512640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f05ac15e8c0 0x7f05ac15cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:41.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05abfff640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0d0140 0x7f05ac0b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05b0d11640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac0cfcb0 0x7f05ac0b8d60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f05a000e960 tx=0x7f05a000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0017070 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a000ca00 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ac0b9740 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a000cdd0 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.076+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ac0b9d00 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.077+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f05ac0689e0 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.077+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f05a001a070 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.078+0000 7f05a97fa640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f059006c980 0x7f059006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.078+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f05a008bd70 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.078+0000 7f05a97fa640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f0590072ec0 0x7f0590075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.078+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 --> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05900759d0 con 0x7f0590072ec0 2024-04-15T15:23:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.078+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f05a008c250 con 0x7f05ac0cfcb0 2024-04-15T15:23:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.078+0000 7f05abfff640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f059006c980 0x7f059006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.078+0000 7f05b1512640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f0590072ec0 0x7f0590075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.078+0000 7f05b1512640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f0590072ec0 0x7f0590075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.079+0000 7f05abfff640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f059006c980 0x7f059006ee40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0598034000 tx=0x7f059802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.079+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 <== osd.2 v2:172.21.15.12:6808/2859338982 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f05900759d0 con 0x7f0590072ec0 2024-04-15T15:23:41.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.092+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 --> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f05ac0b13d0 con 0x7f0590072ec0 2024-04-15T15:23:41.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.092+0000 7f05a97fa640 1 -- 172.21.15.12:0/1183212269 <== osd.2 v2:172.21.15.12:6808/2859338982 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f05ac0b13d0 con 0x7f0590072ec0 2024-04-15T15:23:41.094 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:41.094 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:23:41.094 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f0590072ec0 msgr2=0x7f0590075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.12:6808/2859338982,v1:172.21.15.12:6809/2859338982] conn(0x7f0590072ec0 0x7f0590075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f059006c980 msgr2=0x7f059006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f059006c980 0x7f059006ee40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0598034000 tx=0x7f059802f000 comp rx=0 tx=0).stop 2024-04-15T15:23:41.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac0cfcb0 msgr2=0x7f05ac0b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 --2- 172.21.15.12:0/1183212269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05ac0cfcb0 0x7f05ac0b8d60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f05a000e960 tx=0x7f05a000ee20 comp rx=0 tx=0).stop 2024-04-15T15:23:41.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b1512640 1 -- 172.21.15.12:0/1183212269 reap_dead start 2024-04-15T15:23:41.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 shutdown_connections 2024-04-15T15:23:41.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 >> 172.21.15.12:0/1183212269 conn(0x7f05ac08bf60 msgr2=0x7f05ac0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:41.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 shutdown_connections 2024-04-15T15:23:41.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.093+0000 7f05b2f75640 1 -- 172.21.15.12:0/1183212269 wait complete. 2024-04-15T15:23:41.105 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-15T15:23:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.209+0000 7f7c339f2640 1 Processor -- start 2024-04-15T15:23:41.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.209+0000 7f7c339f2640 1 -- start start 2024-04-15T15:23:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c339f2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7c2c15e700 0x7f7c2c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c339f2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c0d2360 0x7f7c2c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c339f2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c2c0b47a0 con 0x7f7c2c15eae0 2024-04-15T15:23:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c339f2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7c2c0b53b0 con 0x7f7c2c15e700 2024-04-15T15:23:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c339f2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b4a80 con 0x7f7c2c0d2360 2024-04-15T15:23:41.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c31f8f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7c2c15eae0 0x7f7c2c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50370/0 (socket says 172.21.15.12:50370) 2024-04-15T15:23:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c31f8f640 1 -- 172.21.15.12:0/3119525912 learned_addr learned my addr 172.21.15.12:0/3119525912 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c30f8d640 1 --2- 172.21.15.12:0/3119525912 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c0d2360 0x7f7c2c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c3178e640 1 --2- 172.21.15.12:0/3119525912 >> v2:172.21.15.55:3300/0 conn(0x7f7c2c15e700 0x7f7c2c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c23fff640 1 -- 172.21.15.12:0/3119525912 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1230273933 0 0) 0x7f7c2c0b47a0 con 0x7f7c2c15eae0 2024-04-15T15:23:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c23fff640 1 -- 172.21.15.12:0/3119525912 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c18003520 con 0x7f7c2c15eae0 2024-04-15T15:23:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.210+0000 7f7c23fff640 1 -- 172.21.15.12:0/3119525912 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2403439173 0 0) 0x7f7c18003520 con 0x7f7c2c15eae0 2024-04-15T15:23:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.211+0000 7f7c30f8d640 1 -- 172.21.15.12:0/3119525912 >> v2:172.21.15.55:3300/0 conn(0x7f7c2c15e700 msgr2=0x7f7c2c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:41.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.211+0000 7f7c30f8d640 1 --2- 172.21.15.12:0/3119525912 >> v2:172.21.15.55:3300/0 conn(0x7f7c2c15e700 0x7f7c2c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.211+0000 7f7c30f8d640 1 -- 172.21.15.12:0/3119525912 >> v1:172.21.15.12:6789/0 conn(0x7f7c2c15eae0 legacy=0x7f7c2c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.211+0000 7f7c30f8d640 1 -- 172.21.15.12:0/3119525912 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2c0b4d60 con 0x7f7c2c0d2360 2024-04-15T15:23:41.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.211+0000 7f7c3178e640 1 --2- 172.21.15.12:0/3119525912 >> v2:172.21.15.55:3300/0 conn(0x7f7c2c15e700 0x7f7c2c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:41.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.211+0000 7f7c30f8d640 1 --2- 172.21.15.12:0/3119525912 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c0d2360 0x7f7c2c0d2b60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0097a0 tx=0x7f7c1c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e7e73dac5d8584d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c23fff640 1 -- 172.21.15.12:0/3119525912 <== mon.2 v2:172.21.15.55:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c037070 con 0x7f7c2c0d2360 2024-04-15T15:23:41.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c23fff640 1 -- 172.21.15.12:0/3119525912 <== mon.2 v2:172.21.15.55:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1c030510 con 0x7f7c2c0d2360 2024-04-15T15:23:41.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c23fff640 1 -- 172.21.15.12:0/3119525912 <== mon.2 v2:172.21.15.55:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c240019f0 con 0x7f7c2c0d2360 2024-04-15T15:23:41.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c339f2640 1 -- 172.21.15.12:0/3119525912 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c0d2360 msgr2=0x7f7c2c0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c339f2640 1 --2- 172.21.15.12:0/3119525912 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c0d2360 0x7f7c2c0d2b60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0097a0 tx=0x7f7c1c02b790 comp rx=0 tx=0).stop 2024-04-15T15:23:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c339f2640 1 -- 172.21.15.12:0/3119525912 shutdown_connections 2024-04-15T15:23:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c339f2640 1 --2- 172.21.15.12:0/3119525912 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c0d2360 0x7f7c2c0d2b60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c339f2640 1 --2- 172.21.15.12:0/3119525912 >> v2:172.21.15.55:3300/0 conn(0x7f7c2c15e700 0x7f7c2c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c339f2640 1 -- 172.21.15.12:0/3119525912 >> 172.21.15.12:0/3119525912 conn(0x7f7c2c08bf60 msgr2=0x7f7c2c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c339f2640 1 -- 172.21.15.12:0/3119525912 shutdown_connections 2024-04-15T15:23:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.212+0000 7f7c339f2640 1 -- 172.21.15.12:0/3119525912 wait complete. 2024-04-15T15:23:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.214+0000 7f7c339f2640 1 Processor -- start 2024-04-15T15:23:41.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.214+0000 7f7c339f2640 1 -- start start 2024-04-15T15:23:41.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c339f2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c2c0d2360 0x7f7c2c0b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c339f2640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c15e700 0x7f7c2c1619b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c339f2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2c15eae0 0x7f7c2c153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c339f2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b5d80 con 0x7f7c2c0d2360 2024-04-15T15:23:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c339f2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b4450 con 0x7f7c2c15eae0 2024-04-15T15:23:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c339f2640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7c2c0d1d40 con 0x7f7c2c15e700 2024-04-15T15:23:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c2c0d2360 0x7f7c2c0b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c2c0d2360 0x7f7c2c0b8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53918/0 (socket says 172.21.15.12:53918) 2024-04-15T15:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 -- 172.21.15.12:0/1070381104 learned_addr learned my addr 172.21.15.12:0/1070381104 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c31f8f640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2c15eae0 0x7f7c2c153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c30f8d640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c15e700 0x7f7c2c1619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 -- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c15e700 msgr2=0x7f7c2c1619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c15e700 0x7f7c2c1619b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 -- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2c15eae0 msgr2=0x7f7c2c153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2c15eae0 0x7f7c2c153d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 -- 172.21.15.12:0/1070381104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2c154270 con 0x7f7c2c0d2360 2024-04-15T15:23:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c3178e640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c2c0d2360 0x7f7c2c0b8920 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800ec10 tx=0x7f7c2800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c21ffb640 1 -- 172.21.15.12:0/1070381104 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2800cc30 con 0x7f7c2c0d2360 2024-04-15T15:23:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c21ffb640 1 -- 172.21.15.12:0/1070381104 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c2800cdd0 con 0x7f7c2c0d2360 2024-04-15T15:23:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2c0c9530 con 0x7f7c2c0d2360 2024-04-15T15:23:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.216+0000 7f7c21ffb640 1 -- 172.21.15.12:0/1070381104 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28005bb0 con 0x7f7c2c0d2360 2024-04-15T15:23:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.217+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2c0c9af0 con 0x7f7c2c0d2360 2024-04-15T15:23:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.218+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2c0689e0 con 0x7f7c2c0d2360 2024-04-15T15:23:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.218+0000 7f7c21ffb640 1 -- 172.21.15.12:0/1070381104 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7c28006030 con 0x7f7c2c0d2360 2024-04-15T15:23:41.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.219+0000 7f7c21ffb640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c1006c980 0x7f7c1006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.219+0000 7f7c21ffb640 1 -- 172.21.15.12:0/1070381104 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f7c2808bf20 con 0x7f7c2c0d2360 2024-04-15T15:23:41.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.219+0000 7f7c30f8d640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c1006c980 0x7f7c1006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.220+0000 7f7c30f8d640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c1006c980 0x7f7c1006ee40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c034000 tx=0x7f7c1c02f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.223+0000 7f7c21ffb640 1 -- 172.21.15.12:0/1070381104 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c28017020 con 0x7f7c2c0d2360 2024-04-15T15:23:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.423+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f7c2c0b3cc0 con 0x7f7c1006c980 2024-04-15T15:23:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.424+0000 7f7c21ffb640 1 -- 172.21.15.12:0/1070381104 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7c2c0b3cc0 con 0x7f7c1006c980 2024-04-15T15:23:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 2 to deep-scrub 2024-04-15T15:23:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c1006c980 msgr2=0x7f7c1006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c1006c980 0x7f7c1006ee40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c034000 tx=0x7f7c1c02f000 comp rx=0 tx=0).stop 2024-04-15T15:23:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c2c0d2360 msgr2=0x7f7c2c0b8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c2c0d2360 0x7f7c2c0b8920 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800ec10 tx=0x7f7c2800c520 comp rx=0 tx=0).stop 2024-04-15T15:23:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 shutdown_connections 2024-04-15T15:23:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7c1006c980 0x7f7c1006ee40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c2c15eae0 0x7f7c2c153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7c2c15e700 0x7f7c2c1619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 --2- 172.21.15.12:0/1070381104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c2c0d2360 0x7f7c2c0b8920 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.426+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 >> 172.21.15.12:0/1070381104 conn(0x7f7c2c08bf60 msgr2=0x7f7c2c0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.427+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 shutdown_connections 2024-04-15T15:23:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.427+0000 7f7c339f2640 1 -- 172.21.15.12:0/1070381104 wait complete. 2024-04-15T15:23:41.439 INFO:tasks.ceph:Scrubbing osd.3 2024-04-15T15:23:41.439 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:23:41.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.546+0000 7fb98f4ce640 1 Processor -- start 2024-04-15T15:23:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.546+0000 7fb98f4ce640 1 -- start start 2024-04-15T15:23:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98f4ce640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb9880b1520 0x7fb9880b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98f4ce640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 0x7fb988162af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98f4ce640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb988163030 con 0x7fb98815eb70 2024-04-15T15:23:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98f4ce640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb9880d0d90 con 0x7fb9880b1520 2024-04-15T15:23:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98f4ce640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb9880b1a20 con 0x7fb9881626e0 2024-04-15T15:23:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98ca69640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb9880b1520 0x7fb9880b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98ca69640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb9880b1520 0x7fb9880b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53922/0 (socket says 172.21.15.12:53922) 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98ca69640 1 -- 172.21.15.12:0/1771279747 learned_addr learned my addr 172.21.15.12:0/1771279747 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98da6b640 1 --2- 172.21.15.12:0/1771279747 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 0x7fb988162af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98ca69640 1 -- 172.21.15.12:0/1771279747 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 msgr2=0x7fb988162af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98ca69640 1 --2- 172.21.15.12:0/1771279747 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 0x7fb988162af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98ca69640 1 -- 172.21.15.12:0/1771279747 >> v1:172.21.15.55:6789/0 conn(0x7fb98815eb70 legacy=0x7fb98809b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98ca69640 1 -- 172.21.15.12:0/1771279747 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9880d0fe0 con 0x7fb9880b1520 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.547+0000 7fb98da6b640 1 --2- 172.21.15.12:0/1771279747 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 0x7fb988162af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.548+0000 7fb98ca69640 1 --2- 172.21.15.12:0/1771279747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 0x7fb9880b4a90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009870 tx=0x7fb97c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0858b83dd27d81 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.548+0000 7fb987fff640 1 -- 172.21.15.12:0/1771279747 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c035020 con 0x7fb9880b1520 2024-04-15T15:23:41.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.548+0000 7fb987fff640 1 -- 172.21.15.12:0/1771279747 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97c031640 con 0x7fb9880b1520 2024-04-15T15:23:41.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.548+0000 7fb987fff640 1 -- 172.21.15.12:0/1771279747 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c02d040 con 0x7fb9880b1520 2024-04-15T15:23:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.549+0000 7fb98f4ce640 1 -- 172.21.15.12:0/1771279747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 msgr2=0x7fb9880b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.549+0000 7fb98f4ce640 1 --2- 172.21.15.12:0/1771279747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 0x7fb9880b4a90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009870 tx=0x7fb97c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.549+0000 7fb98f4ce640 1 -- 172.21.15.12:0/1771279747 shutdown_connections 2024-04-15T15:23:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.549+0000 7fb98f4ce640 1 --2- 172.21.15.12:0/1771279747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 0x7fb9880b4a90 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.549+0000 7fb98f4ce640 1 --2- 172.21.15.12:0/1771279747 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 0x7fb988162af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.549+0000 7fb98f4ce640 1 -- 172.21.15.12:0/1771279747 >> 172.21.15.12:0/1771279747 conn(0x7fb9880b3f70 msgr2=0x7fb9880b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:41.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.549+0000 7fb98f4ce640 1 -- 172.21.15.12:0/1771279747 shutdown_connections 2024-04-15T15:23:41.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.549+0000 7fb98f4ce640 1 -- 172.21.15.12:0/1771279747 wait complete. 2024-04-15T15:23:41.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.550+0000 7fb98f4ce640 1 Processor -- start 2024-04-15T15:23:41.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.550+0000 7fb98f4ce640 1 -- start start 2024-04-15T15:23:41.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98f4ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 0x7fb98814a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98f4ce640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb98815eb70 0x7fb98815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98f4ce640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 0x7fb9881e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98f4ce640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb98815ff60 con 0x7fb9880b1520 2024-04-15T15:23:41.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98f4ce640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb9881634a0 con 0x7fb98815eb70 2024-04-15T15:23:41.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98f4ce640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fb98806d160 con 0x7fb9881626e0 2024-04-15T15:23:41.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98d26a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 0x7fb98814a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98d26a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 0x7fb98814a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53932/0 (socket says 172.21.15.12:53932) 2024-04-15T15:23:41.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98d26a640 1 -- 172.21.15.12:0/3522377164 learned_addr learned my addr 172.21.15.12:0/3522377164 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:41.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98da6b640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 0x7fb9881e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98ca69640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb98815eb70 0x7fb98815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98d26a640 1 -- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 msgr2=0x7fb9881e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98d26a640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fb9881626e0 0x7fb9881e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98d26a640 1 -- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb98815eb70 msgr2=0x7fb98815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98d26a640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb98815eb70 0x7fb98815c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.551+0000 7fb98d26a640 1 -- 172.21.15.12:0/3522377164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98815cb60 con 0x7fb9880b1520 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.552+0000 7fb98d26a640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 0x7fb98814a9a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb97800c790 tx=0x7fb97800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.552+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978016020 con 0x7fb9880b1520 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.552+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb978006d50 con 0x7fb9880b1520 2024-04-15T15:23:41.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.552+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb98815ce40 con 0x7fb9880b1520 2024-04-15T15:23:41.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.552+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978003bb0 con 0x7fb9880b1520 2024-04-15T15:23:41.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.552+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb98815d400 con 0x7fb9880b1520 2024-04-15T15:23:41.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.553+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9880689e0 con 0x7fb9880b1520 2024-04-15T15:23:41.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.553+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb97800a850 con 0x7fb9880b1520 2024-04-15T15:23:41.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.554+0000 7fb985ffb640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb97406c980 0x7fb97406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.554+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fb97808b5a0 con 0x7fb9880b1520 2024-04-15T15:23:41.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.554+0000 7fb985ffb640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7fb974072ec0 0x7fb974075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.554+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 --> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb97800de50 con 0x7fb974072ec0 2024-04-15T15:23:41.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.554+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb978092050 con 0x7fb9880b1520 2024-04-15T15:23:41.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.554+0000 7fb98ca69640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb97406c980 0x7fb97406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.554+0000 7fb98da6b640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7fb974072ec0 0x7fb974075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.555+0000 7fb98da6b640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7fb974072ec0 0x7fb974075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.555+0000 7fb98ca69640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb97406c980 0x7fb97406ee40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb97c036000 tx=0x7fb97c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.555+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 <== osd.3 v2:172.21.15.12:6824/3520688318 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb97800de50 con 0x7fb974072ec0 2024-04-15T15:23:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.568+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 --> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb9880b8400 con 0x7fb974072ec0 2024-04-15T15:23:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb985ffb640 1 -- 172.21.15.12:0/3522377164 <== osd.3 v2:172.21.15.12:6824/3520688318 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb9880b8400 con 0x7fb974072ec0 2024-04-15T15:23:41.570 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:41.571 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:23:41.571 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7fb974072ec0 msgr2=0x7fb974075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.12:6824/3520688318,v1:172.21.15.12:6825/3520688318] conn(0x7fb974072ec0 0x7fb974075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb97406c980 msgr2=0x7fb97406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fb97406c980 0x7fb97406ee40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb97c036000 tx=0x7fb97c03c040 comp rx=0 tx=0).stop 2024-04-15T15:23:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 msgr2=0x7fb98814a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 --2- 172.21.15.12:0/3522377164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9880b1520 0x7fb98814a9a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb97800c790 tx=0x7fb97800cc50 comp rx=0 tx=0).stop 2024-04-15T15:23:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98da6b640 1 -- 172.21.15.12:0/3522377164 reap_dead start 2024-04-15T15:23:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 shutdown_connections 2024-04-15T15:23:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 >> 172.21.15.12:0/3522377164 conn(0x7fb9880b3f70 msgr2=0x7fb9880ccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 shutdown_connections 2024-04-15T15:23:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.569+0000 7fb98f4ce640 1 -- 172.21.15.12:0/3522377164 wait complete. 2024-04-15T15:23:41.581 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-15T15:23:41.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.685+0000 7f585aec1640 1 Processor -- start 2024-04-15T15:23:41.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.685+0000 7f585aec1640 1 -- start start 2024-04-15T15:23:41.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f585aec1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58540af340 0x7f58540b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f585aec1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58540af920 con 0x7f585415ecb0 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f585aec1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58540afc00 con 0x7f58540b1620 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f585aec1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f585415f090 con 0x7f58540af340 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f584bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58540af340 0x7f58540b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f584bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58540af340 0x7f58540b4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53944/0 (socket says 172.21.15.12:53944) 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f584bfff640 1 -- 172.21.15.12:0/1766631791 learned_addr learned my addr 172.21.15.12:0/1766631791 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f584bfff640 1 -- 172.21.15.12:0/1766631791 >> v1:172.21.15.55:6789/0 conn(0x7f58540b1620 legacy=0x7f58540ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f584bfff640 1 -- 172.21.15.12:0/1766631791 >> v1:172.21.15.12:6789/0 conn(0x7f585415ecb0 legacy=0x7f58540b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.686+0000 7f584bfff640 1 -- 172.21.15.12:0/1766631791 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58540afee0 con 0x7f58540af340 2024-04-15T15:23:41.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.687+0000 7f584bfff640 1 --2- 172.21.15.12:0/1766631791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58540af340 0x7f58540b4350 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5844009870 tx=0x7f584402bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b582ceefe7b768c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.687+0000 7f584b7fe640 1 -- 172.21.15.12:0/1766631791 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584402d870 con 0x7f58540af340 2024-04-15T15:23:41.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.687+0000 7f584b7fe640 1 -- 172.21.15.12:0/1766631791 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f584402da10 con 0x7f58540af340 2024-04-15T15:23:41.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.687+0000 7f584b7fe640 1 -- 172.21.15.12:0/1766631791 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584402dde0 con 0x7f58540af340 2024-04-15T15:23:41.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.688+0000 7f585aec1640 1 -- 172.21.15.12:0/1766631791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58540af340 msgr2=0x7f58540b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.688+0000 7f585aec1640 1 --2- 172.21.15.12:0/1766631791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58540af340 0x7f58540b4350 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5844009870 tx=0x7f584402bab0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.688+0000 7f585aec1640 1 -- 172.21.15.12:0/1766631791 shutdown_connections 2024-04-15T15:23:41.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.688+0000 7f585aec1640 1 --2- 172.21.15.12:0/1766631791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58540af340 0x7f58540b4350 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.688+0000 7f585aec1640 1 -- 172.21.15.12:0/1766631791 >> 172.21.15.12:0/1766631791 conn(0x7f58540b6870 msgr2=0x7f58540b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:41.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.688+0000 7f585aec1640 1 -- 172.21.15.12:0/1766631791 shutdown_connections 2024-04-15T15:23:41.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.688+0000 7f585aec1640 1 -- 172.21.15.12:0/1766631791 wait complete. 2024-04-15T15:23:41.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.689+0000 7f585aec1640 1 Processor -- start 2024-04-15T15:23:41.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.689+0000 7f585aec1640 1 -- start start 2024-04-15T15:23:41.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585aec1640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58540af340 0x7f58541549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585aec1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58540b1620 0x7f5854161970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585aec1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585415ecb0 0x7f58540cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585aec1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f58540ceb20 con 0x7f585415ecb0 2024-04-15T15:23:41.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585aec1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f58540cdf80 con 0x7f58540b1620 2024-04-15T15:23:41.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585aec1640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f58540ceca0 con 0x7f58540af340 2024-04-15T15:23:41.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585415ecb0 0x7f58540cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585415ecb0 0x7f58540cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53954/0 (socket says 172.21.15.12:53954) 2024-04-15T15:23:41.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 -- 172.21.15.12:0/225296094 learned_addr learned my addr 172.21.15.12:0/225296094 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:41.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f584bfff640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58540b1620 0x7f5854161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f5858c5d640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58540af340 0x7f58541549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 -- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58540af340 msgr2=0x7f58541549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58540af340 0x7f58541549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 -- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58540b1620 msgr2=0x7f5854161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58540b1620 0x7f5854161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 -- 172.21.15.12:0/225296094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5854159210 con 0x7f585415ecb0 2024-04-15T15:23:41.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.690+0000 7f585945e640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585415ecb0 0x7f58540cc0d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f584c00c730 tx=0x7f584c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.691+0000 7f58497fa640 1 -- 172.21.15.12:0/225296094 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c003b70 con 0x7f585415ecb0 2024-04-15T15:23:41.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.691+0000 7f58497fa640 1 -- 172.21.15.12:0/225296094 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f584c003d30 con 0x7f585415ecb0 2024-04-15T15:23:41.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.691+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5854160090 con 0x7f585415ecb0 2024-04-15T15:23:41.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.691+0000 7f58497fa640 1 -- 172.21.15.12:0/225296094 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c005310 con 0x7f585415ecb0 2024-04-15T15:23:41.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.691+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5854160650 con 0x7f585415ecb0 2024-04-15T15:23:41.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.692+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58541593e0 con 0x7f585415ecb0 2024-04-15T15:23:41.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.692+0000 7f58497fa640 1 -- 172.21.15.12:0/225296094 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f584c0056c0 con 0x7f585415ecb0 2024-04-15T15:23:41.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.692+0000 7f58497fa640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f584006c980 0x7f584006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:41.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.693+0000 7f58497fa640 1 -- 172.21.15.12:0/225296094 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f584c08b9b0 con 0x7f585415ecb0 2024-04-15T15:23:41.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.693+0000 7f5858c5d640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f584006c980 0x7f584006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:41.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.693+0000 7f5858c5d640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f584006c980 0x7f584006ee40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f58541f80a0 tx=0x7f5850009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:41.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.697+0000 7f58497fa640 1 -- 172.21.15.12:0/225296094 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f584c0598c0 con 0x7f585415ecb0 2024-04-15T15:23:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.882+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f58540689e0 con 0x7f584006c980 2024-04-15T15:23:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.883+0000 7f58497fa640 1 -- 172.21.15.12:0/225296094 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f58540689e0 con 0x7f584006c980 2024-04-15T15:23:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 3 to deep-scrub 2024-04-15T15:23:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f584006c980 msgr2=0x7f584006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f584006c980 0x7f584006ee40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f58541f80a0 tx=0x7f5850009040 comp rx=0 tx=0).stop 2024-04-15T15:23:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585415ecb0 msgr2=0x7f58540cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585415ecb0 0x7f58540cc0d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f584c00c730 tx=0x7f584c00cbf0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 shutdown_connections 2024-04-15T15:23:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585415ecb0 0x7f58540cc0d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f584006c980 0x7f584006ee40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58540b1620 0x7f5854161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 --2- 172.21.15.12:0/225296094 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f58540af340 0x7f58541549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 >> 172.21.15.12:0/225296094 conn(0x7f58540b6870 msgr2=0x7f58540b4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 shutdown_connections 2024-04-15T15:23:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:41.885+0000 7f585aec1640 1 -- 172.21.15.12:0/225296094 wait complete. 2024-04-15T15:23:41.897 INFO:tasks.ceph:Scrubbing osd.4 2024-04-15T15:23:41.897 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:23:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.004+0000 7fa2fa263640 1 Processor -- start 2024-04-15T15:23:42.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.004+0000 7fa2fa263640 1 -- start start 2024-04-15T15:23:42.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2fa263640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa2f415eb70 0x7fa2f40d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2fa263640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2f40d1070 0x7fa2f40d14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2fa263640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2f40d2c00 con 0x7fa2f40cfe80 2024-04-15T15:23:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2fa263640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f406d7c0 con 0x7fa2f40d1070 2024-04-15T15:23:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2fa263640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f40d1a20 con 0x7fa2f415eb70 2024-04-15T15:23:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2f8800640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2f40d1070 0x7fa2f40d14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2f8800640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa2f40d1070 0x7fa2f40d14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53966/0 (socket says 172.21.15.12:53966) 2024-04-15T15:23:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2f8800640 1 -- 172.21.15.12:0/1571741691 learned_addr learned my addr 172.21.15.12:0/1571741691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2f3fff640 1 --2- 172.21.15.12:0/1571741691 >> v2:172.21.15.55:3300/0 conn(0x7fa2f415eb70 0x7fa2f40d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2f8800640 1 -- 172.21.15.12:0/1571741691 >> v2:172.21.15.55:3300/0 conn(0x7fa2f415eb70 msgr2=0x7fa2f40d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2f8800640 1 --2- 172.21.15.12:0/1571741691 >> v2:172.21.15.55:3300/0 conn(0x7fa2f415eb70 0x7fa2f40d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2f8800640 1 -- 172.21.15.12:0/1571741691 >> v1:172.21.15.12:6789/0 conn(0x7fa2f40cfe80 legacy=0x7fa2f406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.005+0000 7fa2f8800640 1 -- 172.21.15.12:0/1571741691 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f40d2ee0 con 0x7fa2f40d1070 2024-04-15T15:23:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.006+0000 7fa2f8800640 1 --2- 172.21.15.12:0/1571741691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 0x7fa2f40d14e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa2f408cf00 tx=0x7fa2ec02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e635330fc7d019 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.006+0000 7fa2f2ffd640 1 -- 172.21.15.12:0/1571741691 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec02cb20 con 0x7fa2f40d1070 2024-04-15T15:23:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.006+0000 7fa2f2ffd640 1 -- 172.21.15.12:0/1571741691 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec02ccc0 con 0x7fa2f40d1070 2024-04-15T15:23:42.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.006+0000 7fa2f2ffd640 1 -- 172.21.15.12:0/1571741691 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec034610 con 0x7fa2f40d1070 2024-04-15T15:23:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.007+0000 7fa2fa263640 1 -- 172.21.15.12:0/1571741691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 msgr2=0x7fa2f40d14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.007+0000 7fa2fa263640 1 --2- 172.21.15.12:0/1571741691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 0x7fa2f40d14e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa2f408cf00 tx=0x7fa2ec02bf80 comp rx=0 tx=0).stop 2024-04-15T15:23:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.007+0000 7fa2fa263640 1 -- 172.21.15.12:0/1571741691 shutdown_connections 2024-04-15T15:23:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.007+0000 7fa2fa263640 1 --2- 172.21.15.12:0/1571741691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 0x7fa2f40d14e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.007+0000 7fa2fa263640 1 --2- 172.21.15.12:0/1571741691 >> v2:172.21.15.55:3300/0 conn(0x7fa2f415eb70 0x7fa2f40d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.007+0000 7fa2fa263640 1 -- 172.21.15.12:0/1571741691 >> 172.21.15.12:0/1571741691 conn(0x7fa2f40b61d0 msgr2=0x7fa2f40b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.007+0000 7fa2fa263640 1 -- 172.21.15.12:0/1571741691 shutdown_connections 2024-04-15T15:23:42.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.007+0000 7fa2fa263640 1 -- 172.21.15.12:0/1571741691 wait complete. 2024-04-15T15:23:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.008+0000 7fa2fa263640 1 Processor -- start 2024-04-15T15:23:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.008+0000 7fa2fa263640 1 -- start start 2024-04-15T15:23:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2fa263640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa2f40cfe80 0x7fa2f4160ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2fa263640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 0x7fa2f41582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2fa263640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f415eb70 0x7fa2f4148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2fa263640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f415ef50 con 0x7fa2f40d1070 2024-04-15T15:23:42.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2fa263640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f40d34a0 con 0x7fa2f415eb70 2024-04-15T15:23:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2fa263640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f40b7b40 con 0x7fa2f40cfe80 2024-04-15T15:23:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 0x7fa2f41582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 0x7fa2f41582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53968/0 (socket says 172.21.15.12:53968) 2024-04-15T15:23:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f37fe640 1 -- 172.21.15.12:0/2699272803 learned_addr learned my addr 172.21.15.12:0/2699272803 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f3fff640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa2f40cfe80 0x7fa2f4160ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f8800640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f415eb70 0x7fa2f4148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f37fe640 1 -- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa2f40cfe80 msgr2=0x7fa2f4160ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f37fe640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fa2f40cfe80 0x7fa2f4160ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f37fe640 1 -- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f415eb70 msgr2=0x7fa2f4148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f37fe640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2f415eb70 0x7fa2f4148cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.009+0000 7fa2f37fe640 1 -- 172.21.15.12:0/2699272803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f4158a10 con 0x7fa2f40d1070 2024-04-15T15:23:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.010+0000 7fa2f37fe640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 0x7fa2f41582b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800c400 tx=0x7fa2e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.010+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80052d0 con 0x7fa2f40d1070 2024-04-15T15:23:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.010+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e8005490 con 0x7fa2f40d1070 2024-04-15T15:23:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.010+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f4158cf0 con 0x7fa2f40d1070 2024-04-15T15:23:42.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.010+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8007260 con 0x7fa2f40d1070 2024-04-15T15:23:42.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.010+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f41592b0 con 0x7fa2f40d1070 2024-04-15T15:23:42.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.011+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa2f408c2d0 con 0x7fa2f40d1070 2024-04-15T15:23:42.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.011+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa2e8007400 con 0x7fa2f40d1070 2024-04-15T15:23:42.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.012+0000 7fa2f0ff9640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa2e006c9b0 0x7fa2e006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.012+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fa2e808b070 con 0x7fa2f40d1070 2024-04-15T15:23:42.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.012+0000 7fa2f0ff9640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7fa2e0072ef0 0x7fa2e0075330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.012+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 --> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2e8009e50 con 0x7fa2e0072ef0 2024-04-15T15:23:42.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.012+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fa2e8092050 con 0x7fa2f40d1070 2024-04-15T15:23:42.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.012+0000 7fa2f3fff640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa2e006c9b0 0x7fa2e006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.012+0000 7fa2f8800640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7fa2e0072ef0 0x7fa2e0075330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.013+0000 7fa2f3fff640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa2e006c9b0 0x7fa2e006ee70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4006fd0 tx=0x7fa2e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.013+0000 7fa2f8800640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7fa2e0072ef0 0x7fa2e0075330 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.013+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 <== osd.4 v2:172.21.15.55:6808/3874250091 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa2e8009e50 con 0x7fa2e0072ef0 2024-04-15T15:23:42.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.026+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 --> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa2f40689e0 con 0x7fa2e0072ef0 2024-04-15T15:23:42.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.027+0000 7fa2f0ff9640 1 -- 172.21.15.12:0/2699272803 <== osd.4 v2:172.21.15.55:6808/3874250091 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa2f40689e0 con 0x7fa2e0072ef0 2024-04-15T15:23:42.029 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:42.029 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:23:42.029 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7fa2e0072ef0 msgr2=0x7fa2e0075330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6808/3874250091,v1:172.21.15.55:6809/3874250091] conn(0x7fa2e0072ef0 0x7fa2e0075330 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa2e006c9b0 msgr2=0x7fa2e006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fa2e006c9b0 0x7fa2e006ee70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4006fd0 tx=0x7fa2e4008040 comp rx=0 tx=0).stop 2024-04-15T15:23:42.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 msgr2=0x7fa2f41582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 --2- 172.21.15.12:0/2699272803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2f40d1070 0x7fa2f41582b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800c400 tx=0x7fa2e800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2f8800640 1 -- 172.21.15.12:0/2699272803 reap_dead start 2024-04-15T15:23:42.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 shutdown_connections 2024-04-15T15:23:42.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 >> 172.21.15.12:0/2699272803 conn(0x7fa2f40b61d0 msgr2=0x7fa2f40b8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 shutdown_connections 2024-04-15T15:23:42.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.028+0000 7fa2fa263640 1 -- 172.21.15.12:0/2699272803 wait complete. 2024-04-15T15:23:42.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-15T15:23:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.144+0000 7f7d4462a640 1 Processor -- start 2024-04-15T15:23:42.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.144+0000 7f7d4462a640 1 -- start start 2024-04-15T15:23:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d4462a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0d0690 0x7f7d3c0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d4462a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7d3c0b3160 0x7f7d3c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d4462a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7d3c15ef70 0x7f7d3c15f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d4462a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7d3c0d0fb0 con 0x7f7d3c0b3160 2024-04-15T15:23:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d4462a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7d3c15cb50 con 0x7f7d3c15ef70 2024-04-15T15:23:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d4462a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3c15bfd0 con 0x7f7d3c0d0690 2024-04-15T15:23:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d41bc5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7d3c0b3160 0x7f7d3c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d41bc5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7d3c0b3160 0x7f7d3c15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53972/0 (socket says 172.21.15.12:53972) 2024-04-15T15:23:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d41bc5640 1 -- 172.21.15.12:0/3823819913 learned_addr learned my addr 172.21.15.12:0/3823819913 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d423c6640 1 --2- 172.21.15.12:0/3823819913 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0d0690 0x7f7d3c0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d42bc7640 1 --2- 172.21.15.12:0/3823819913 >> v2:172.21.15.55:3300/0 conn(0x7f7d3c15ef70 0x7f7d3c15f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d41bc5640 1 -- 172.21.15.12:0/3823819913 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0d0690 msgr2=0x7f7d3c0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d41bc5640 1 --2- 172.21.15.12:0/3823819913 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0d0690 0x7f7d3c0d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d41bc5640 1 -- 172.21.15.12:0/3823819913 >> v2:172.21.15.55:3300/0 conn(0x7f7d3c15ef70 msgr2=0x7f7d3c15f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d41bc5640 1 --2- 172.21.15.12:0/3823819913 >> v2:172.21.15.55:3300/0 conn(0x7f7d3c15ef70 0x7f7d3c15f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d41bc5640 1 -- 172.21.15.12:0/3823819913 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c15c2b0 con 0x7f7d3c0b3160 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.145+0000 7f7d42bc7640 1 --2- 172.21.15.12:0/3823819913 >> v2:172.21.15.55:3300/0 conn(0x7f7d3c15ef70 0x7f7d3c15f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:42.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.146+0000 7f7d41bc5640 1 --2- 172.21.15.12:0/3823819913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c0b3160 0x7f7d3c15b690 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0b4ba0 tx=0x7f7d3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e59c29e949adc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.146+0000 7f7d413c4640 1 -- 172.21.15.12:0/3823819913 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d30035020 con 0x7f7d3c0b3160 2024-04-15T15:23:42.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.146+0000 7f7d413c4640 1 -- 172.21.15.12:0/3823819913 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d30031870 con 0x7f7d3c0b3160 2024-04-15T15:23:42.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.146+0000 7f7d413c4640 1 -- 172.21.15.12:0/3823819913 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d30031c40 con 0x7f7d3c0b3160 2024-04-15T15:23:42.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 -- 172.21.15.12:0/3823819913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c0b3160 msgr2=0x7f7d3c15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 --2- 172.21.15.12:0/3823819913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c0b3160 0x7f7d3c15b690 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0b4ba0 tx=0x7f7d3002b910 comp rx=0 tx=0).stop 2024-04-15T15:23:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 -- 172.21.15.12:0/3823819913 shutdown_connections 2024-04-15T15:23:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 --2- 172.21.15.12:0/3823819913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c0b3160 0x7f7d3c15b690 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 --2- 172.21.15.12:0/3823819913 >> v2:172.21.15.55:3300/0 conn(0x7f7d3c15ef70 0x7f7d3c15f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 --2- 172.21.15.12:0/3823819913 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0d0690 0x7f7d3c0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 -- 172.21.15.12:0/3823819913 >> 172.21.15.12:0/3823819913 conn(0x7f7d3c06be00 msgr2=0x7f7d3c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 -- 172.21.15.12:0/3823819913 shutdown_connections 2024-04-15T15:23:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.147+0000 7f7d4462a640 1 -- 172.21.15.12:0/3823819913 wait complete. 2024-04-15T15:23:42.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.148+0000 7f7d4462a640 1 Processor -- start 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.148+0000 7f7d4462a640 1 -- start start 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d4462a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0b3160 0x7f7d3c099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d4462a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d3c0d0690 0x7f7d3c1432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d4462a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c15ef70 0x7f7d3c1458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d4462a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c0b4fb0 con 0x7f7d3c15ef70 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d4462a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c15c870 con 0x7f7d3c0d0690 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d4462a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3c15bbd0 con 0x7f7d3c0b3160 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d42bc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c15ef70 0x7f7d3c1458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d42bc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c15ef70 0x7f7d3c1458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53974/0 (socket says 172.21.15.12:53974) 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d42bc7640 1 -- 172.21.15.12:0/715769941 learned_addr learned my addr 172.21.15.12:0/715769941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d41bc5640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d3c0d0690 0x7f7d3c1432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.149+0000 7f7d423c6640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0b3160 0x7f7d3c099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d42bc7640 1 -- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0b3160 msgr2=0x7f7d3c099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d42bc7640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0b3160 0x7f7d3c099a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d42bc7640 1 -- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d3c0d0690 msgr2=0x7f7d3c1432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d42bc7640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d3c0d0690 0x7f7d3c1432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d42bc7640 1 -- 172.21.15.12:0/715769941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c15cf10 con 0x7f7d3c15ef70 2024-04-15T15:23:42.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d42bc7640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c15ef70 0x7f7d3c1458e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c007f20 tx=0x7f7d2c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d3affd640 1 -- 172.21.15.12:0/715769941 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c01a070 con 0x7f7d3c15ef70 2024-04-15T15:23:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d3affd640 1 -- 172.21.15.12:0/715769941 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2c003b90 con 0x7f7d3c15ef70 2024-04-15T15:23:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3c15d1f0 con 0x7f7d3c15ef70 2024-04-15T15:23:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d3affd640 1 -- 172.21.15.12:0/715769941 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c004030 con 0x7f7d3c15ef70 2024-04-15T15:23:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.150+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3c15d7b0 con 0x7f7d3c15ef70 2024-04-15T15:23:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.151+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d3c0689e0 con 0x7f7d3c15ef70 2024-04-15T15:23:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.151+0000 7f7d3affd640 1 -- 172.21.15.12:0/715769941 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7d2c01d070 con 0x7f7d3c15ef70 2024-04-15T15:23:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.152+0000 7f7d3affd640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d2806c980 0x7f7d2806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.152+0000 7f7d3affd640 1 -- 172.21.15.12:0/715769941 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f7d2c08bd40 con 0x7f7d3c15ef70 2024-04-15T15:23:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.152+0000 7f7d423c6640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d2806c980 0x7f7d2806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.153+0000 7f7d423c6640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d2806c980 0x7f7d2806ee40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009a20 tx=0x7f7d34009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.156+0000 7f7d3affd640 1 -- 172.21.15.12:0/715769941 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d2c010070 con 0x7f7d3c15ef70 2024-04-15T15:23:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.342+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f7d3c0739e0 con 0x7f7d2806c980 2024-04-15T15:23:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.343+0000 7f7d3affd640 1 -- 172.21.15.12:0/715769941 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7d3c0739e0 con 0x7f7d2806c980 2024-04-15T15:23:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 4 to deep-scrub 2024-04-15T15:23:42.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d2806c980 msgr2=0x7f7d2806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d2806c980 0x7f7d2806ee40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009a20 tx=0x7f7d34009190 comp rx=0 tx=0).stop 2024-04-15T15:23:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c15ef70 msgr2=0x7f7d3c1458e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c15ef70 0x7f7d3c1458e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c007f20 tx=0x7f7d2c00d560 comp rx=0 tx=0).stop 2024-04-15T15:23:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 shutdown_connections 2024-04-15T15:23:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d3c15ef70 0x7f7d3c1458e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f7d2806c980 0x7f7d2806ee40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d3c0d0690 0x7f7d3c1432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 --2- 172.21.15.12:0/715769941 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f7d3c0b3160 0x7f7d3c099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 >> 172.21.15.12:0/715769941 conn(0x7f7d3c06be00 msgr2=0x7f7d3c15a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 shutdown_connections 2024-04-15T15:23:42.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.345+0000 7f7d4462a640 1 -- 172.21.15.12:0/715769941 wait complete. 2024-04-15T15:23:42.357 INFO:tasks.ceph:Scrubbing osd.5 2024-04-15T15:23:42.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:23:42.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.464+0000 7fba6c9ba640 1 Processor -- start 2024-04-15T15:23:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.464+0000 7fba6c9ba640 1 -- start start 2024-04-15T15:23:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba6c9ba640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba680b4170 0x7fba68162d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba6c9ba640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba680ae000 con 0x7fba6815e540 2024-04-15T15:23:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba6c9ba640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba680b48a0 con 0x7fba680b2410 2024-04-15T15:23:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba6c9ba640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fba6806e5f0 con 0x7fba680b4170 2024-04-15T15:23:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba5dd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba680b4170 0x7fba68162d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba5dd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba680b4170 0x7fba68162d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37436/0 (socket says 172.21.15.12:37436) 2024-04-15T15:23:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba5dd9b640 1 -- 172.21.15.12:0/184479814 learned_addr learned my addr 172.21.15.12:0/184479814 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba5dd9b640 1 -- 172.21.15.12:0/184479814 >> v1:172.21.15.55:6789/0 conn(0x7fba680b2410 legacy=0x7fba6806f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba5dd9b640 1 -- 172.21.15.12:0/184479814 >> v1:172.21.15.12:6789/0 conn(0x7fba6815e540 legacy=0x7fba680ad8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.465+0000 7fba5dd9b640 1 -- 172.21.15.12:0/184479814 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6806e8d0 con 0x7fba680b4170 2024-04-15T15:23:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.466+0000 7fba5dd9b640 1 --2- 172.21.15.12:0/184479814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 0x7fba68162d10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fba50009870 tx=0x7fba5002bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7556be01ce4fb32f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.466+0000 7fba65d9b640 1 -- 172.21.15.12:0/184479814 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5002d870 con 0x7fba680b4170 2024-04-15T15:23:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.466+0000 7fba65d9b640 1 -- 172.21.15.12:0/184479814 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5002da10 con 0x7fba680b4170 2024-04-15T15:23:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.466+0000 7fba65d9b640 1 -- 172.21.15.12:0/184479814 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5002dde0 con 0x7fba680b4170 2024-04-15T15:23:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.467+0000 7fba6c9ba640 1 -- 172.21.15.12:0/184479814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 msgr2=0x7fba68162d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.467+0000 7fba6c9ba640 1 --2- 172.21.15.12:0/184479814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 0x7fba68162d10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fba50009870 tx=0x7fba5002bab0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.467+0000 7fba6c9ba640 1 -- 172.21.15.12:0/184479814 shutdown_connections 2024-04-15T15:23:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.467+0000 7fba6c9ba640 1 --2- 172.21.15.12:0/184479814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 0x7fba68162d10 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.467+0000 7fba6c9ba640 1 -- 172.21.15.12:0/184479814 >> 172.21.15.12:0/184479814 conn(0x7fba6806b9e0 msgr2=0x7fba6806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.467+0000 7fba6c9ba640 1 -- 172.21.15.12:0/184479814 shutdown_connections 2024-04-15T15:23:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.467+0000 7fba6c9ba640 1 -- 172.21.15.12:0/184479814 wait complete. 2024-04-15T15:23:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.468+0000 7fba6c9ba640 1 Processor -- start 2024-04-15T15:23:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.468+0000 7fba6c9ba640 1 -- start start 2024-04-15T15:23:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba6c9ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba680b2410 0x7fba681e8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba6c9ba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 0x7fba680b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba6c9ba640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fba6815e540 0x7fba681427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba6c9ba640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba680b45b0 con 0x7fba680b4170 2024-04-15T15:23:42.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba6c9ba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fba6806ee90 con 0x7fba680b2410 2024-04-15T15:23:42.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba6c9ba640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fba6815e920 con 0x7fba6815e540 2024-04-15T15:23:42.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba5dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 0x7fba680b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba5dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 0x7fba680b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37440/0 (socket says 172.21.15.12:37440) 2024-04-15T15:23:42.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba5dd9b640 1 -- 172.21.15.12:0/1588915808 learned_addr learned my addr 172.21.15.12:0/1588915808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba6659c640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba680b2410 0x7fba681e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba66d9d640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fba6815e540 0x7fba681427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba5dd9b640 1 -- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fba6815e540 msgr2=0x7fba681427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba5dd9b640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fba6815e540 0x7fba681427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba5dd9b640 1 -- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba680b2410 msgr2=0x7fba681e8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba5dd9b640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba680b2410 0x7fba681e8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.469+0000 7fba5dd9b640 1 -- 172.21.15.12:0/1588915808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba680b1130 con 0x7fba680b4170 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.470+0000 7fba5dd9b640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 0x7fba680b08e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fba5002baf0 tx=0x7fba50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.470+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50047070 con 0x7fba680b4170 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.470+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba50037ce0 con 0x7fba680b4170 2024-04-15T15:23:42.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.470+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5002e680 con 0x7fba680b4170 2024-04-15T15:23:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.470+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba680b1410 con 0x7fba680b4170 2024-04-15T15:23:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.470+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba681433e0 con 0x7fba680b4170 2024-04-15T15:23:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.471+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba680689e0 con 0x7fba680b4170 2024-04-15T15:23:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.471+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fba5002d890 con 0x7fba680b4170 2024-04-15T15:23:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.472+0000 7fba5f7fe640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fba4806c980 0x7fba4806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.472+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fba500ad900 con 0x7fba680b4170 2024-04-15T15:23:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.472+0000 7fba5f7fe640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7fba48072ec0 0x7fba48075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.472+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 --> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba480759d0 con 0x7fba48072ec0 2024-04-15T15:23:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.472+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fba500b22a0 con 0x7fba680b4170 2024-04-15T15:23:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.472+0000 7fba6659c640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fba4806c980 0x7fba4806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.472+0000 7fba66d9d640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7fba48072ec0 0x7fba48075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.473+0000 7fba6659c640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fba4806c980 0x7fba4806ee40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fba58000f60 tx=0x7fba58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.473+0000 7fba66d9d640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7fba48072ec0 0x7fba48075300 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.473+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 <== osd.5 v2:172.21.15.55:6824/4105274290 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba480759d0 con 0x7fba48072ec0 2024-04-15T15:23:42.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.486+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 --> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fba6815eef0 con 0x7fba48072ec0 2024-04-15T15:23:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba5f7fe640 1 -- 172.21.15.12:0/1588915808 <== osd.5 v2:172.21.15.55:6824/4105274290 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fba6815eef0 con 0x7fba48072ec0 2024-04-15T15:23:42.488 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:42.489 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:23:42.489 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7fba48072ec0 msgr2=0x7fba48075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6824/4105274290,v1:172.21.15.55:6825/4105274290] conn(0x7fba48072ec0 0x7fba48075300 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fba4806c980 msgr2=0x7fba4806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fba4806c980 0x7fba4806ee40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fba58000f60 tx=0x7fba58009040 comp rx=0 tx=0).stop 2024-04-15T15:23:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 msgr2=0x7fba680b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 --2- 172.21.15.12:0/1588915808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba680b4170 0x7fba680b08e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fba5002baf0 tx=0x7fba50004730 comp rx=0 tx=0).stop 2024-04-15T15:23:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba66d9d640 1 -- 172.21.15.12:0/1588915808 reap_dead start 2024-04-15T15:23:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 shutdown_connections 2024-04-15T15:23:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 >> 172.21.15.12:0/1588915808 conn(0x7fba6806b9e0 msgr2=0x7fba680ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 shutdown_connections 2024-04-15T15:23:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.487+0000 7fba6c9ba640 1 -- 172.21.15.12:0/1588915808 wait complete. 2024-04-15T15:23:42.499 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-15T15:23:42.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.604+0000 7f68a0b7c640 1 Processor -- start 2024-04-15T15:23:42.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.604+0000 7f68a0b7c640 1 -- start start 2024-04-15T15:23:42.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.604+0000 7f68a0b7c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f689c070940 0x7f689c09b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.604+0000 7f68a0b7c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c0d0da0 0x7f689c0d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.604+0000 7f68a0b7c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f689c0d05e0 con 0x7f689c160310 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.604+0000 7f68a0b7c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f689c15f720 con 0x7f689c070940 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.604+0000 7f68a0b7c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f689c15fa00 con 0x7f689c0d0da0 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.605+0000 7f689a59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f689c070940 0x7f689c09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.605+0000 7f6893fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f689c160310 0x7f689c070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:33160/0 (socket says 172.21.15.12:33160) 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.605+0000 7f6893fff640 1 -- 172.21.15.12:0/3642795474 learned_addr learned my addr 172.21.15.12:0/3642795474 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.605+0000 7f689ad9d640 1 --2- 172.21.15.12:0/3642795474 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c0d0da0 0x7f689c0d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.605+0000 7f689a59c640 1 -- 172.21.15.12:0/3642795474 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c0d0da0 msgr2=0x7f689c0d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:42.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.605+0000 7f689a59c640 1 --2- 172.21.15.12:0/3642795474 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c0d0da0 0x7f689c0d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.605+0000 7f689a59c640 1 -- 172.21.15.12:0/3642795474 >> v1:172.21.15.55:6789/0 conn(0x7f689c160310 legacy=0x7f689c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.605+0000 7f689a59c640 1 -- 172.21.15.12:0/3642795474 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689c15fce0 con 0x7f689c070940 2024-04-15T15:23:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.606+0000 7f689a59c640 1 --2- 172.21.15.12:0/3642795474 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c070940 0x7f689c09b880 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f688c0097a0 tx=0x7f688c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7289d56da41077f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.606+0000 7f6899d9b640 1 -- 172.21.15.12:0/3642795474 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c035020 con 0x7f689c070940 2024-04-15T15:23:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.606+0000 7f6899d9b640 1 -- 172.21.15.12:0/3642795474 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f688c031830 con 0x7f689c070940 2024-04-15T15:23:42.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.606+0000 7f6899d9b640 1 -- 172.21.15.12:0/3642795474 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c031c00 con 0x7f689c070940 2024-04-15T15:23:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.607+0000 7f68a0b7c640 1 -- 172.21.15.12:0/3642795474 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c070940 msgr2=0x7f689c09b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.607+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/3642795474 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c070940 0x7f689c09b880 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f688c0097a0 tx=0x7f688c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.607+0000 7f68a0b7c640 1 -- 172.21.15.12:0/3642795474 shutdown_connections 2024-04-15T15:23:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.607+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/3642795474 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c0d0da0 0x7f689c0d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.607+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/3642795474 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c070940 0x7f689c09b880 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.607+0000 7f68a0b7c640 1 -- 172.21.15.12:0/3642795474 >> 172.21.15.12:0/3642795474 conn(0x7f689c0b3f40 msgr2=0x7f689c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.607+0000 7f68a0b7c640 1 -- 172.21.15.12:0/3642795474 shutdown_connections 2024-04-15T15:23:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.607+0000 7f68a0b7c640 1 -- 172.21.15.12:0/3642795474 wait complete. 2024-04-15T15:23:42.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.608+0000 7f68a0b7c640 1 Processor -- start 2024-04-15T15:23:42.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.608+0000 7f68a0b7c640 1 -- start start 2024-04-15T15:23:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f68a0b7c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f689c070940 0x7f689c14d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f68a0b7c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c0d0da0 0x7f689c0b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f68a0b7c640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c160310 0x7f689c0c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f68a0b7c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f689c0d1770 con 0x7f689c070940 2024-04-15T15:23:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f68a0b7c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f689c160770 con 0x7f689c0d0da0 2024-04-15T15:23:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f68a0b7c640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f689c0d18f0 con 0x7f689c160310 2024-04-15T15:23:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f689c070940 0x7f689c14d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f689c070940 0x7f689c14d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37452/0 (socket says 172.21.15.12:37452) 2024-04-15T15:23:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 -- 172.21.15.12:0/523780455 learned_addr learned my addr 172.21.15.12:0/523780455 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f6893fff640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c0d0da0 0x7f689c0b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689ad9d640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c160310 0x7f689c0c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 -- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c160310 msgr2=0x7f689c0c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c160310 0x7f689c0c7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 -- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c0d0da0 msgr2=0x7f689c0b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c0d0da0 0x7f689c0b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 -- 172.21.15.12:0/523780455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689c0c5fb0 con 0x7f689c070940 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.609+0000 7f689a59c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f689c070940 0x7f689c14d910 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f688c034040 tx=0x7f688c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.610+0000 7f6892ffd640 1 -- 172.21.15.12:0/523780455 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c040070 con 0x7f689c070940 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.610+0000 7f6892ffd640 1 -- 172.21.15.12:0/523780455 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f688c03e070 con 0x7f689c070940 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.610+0000 7f6892ffd640 1 -- 172.21.15.12:0/523780455 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c035040 con 0x7f689c070940 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.610+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689c0b6700 con 0x7f689c070940 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.610+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f689c0b7e90 con 0x7f689c070940 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.611+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f689c0689e0 con 0x7f689c070940 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.611+0000 7f6892ffd640 1 -- 172.21.15.12:0/523780455 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f688c038020 con 0x7f689c070940 2024-04-15T15:23:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.611+0000 7f6892ffd640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f687c06c980 0x7f687c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.612+0000 7f6892ffd640 1 -- 172.21.15.12:0/523780455 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f688c0ad2e0 con 0x7f689c070940 2024-04-15T15:23:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.612+0000 7f6893fff640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f687c06c980 0x7f687c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.612+0000 7f6893fff640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f687c06c980 0x7f687c06ee40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f689c15e610 tx=0x7f6884007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.616+0000 7f6892ffd640 1 -- 172.21.15.12:0/523780455 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f688c031800 con 0x7f689c070940 2024-04-15T15:23:42.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.803+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f689c0b6c20 con 0x7f687c06c980 2024-04-15T15:23:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.803+0000 7f6892ffd640 1 -- 172.21.15.12:0/523780455 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6884001460 con 0x7f687c06c980 2024-04-15T15:23:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 5 to deep-scrub 2024-04-15T15:23:42.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f687c06c980 msgr2=0x7f687c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f687c06c980 0x7f687c06ee40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f689c15e610 tx=0x7f6884007e80 comp rx=0 tx=0).stop 2024-04-15T15:23:42.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f689c070940 msgr2=0x7f689c14d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f689c070940 0x7f689c14d910 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f688c034040 tx=0x7f688c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 shutdown_connections 2024-04-15T15:23:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f689c160310 0x7f689c0c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f687c06c980 0x7f687c06ee40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f689c0d0da0 0x7f689c0b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 --2- 172.21.15.12:0/523780455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f689c070940 0x7f689c14d910 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.805+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 >> 172.21.15.12:0/523780455 conn(0x7f689c0b3f40 msgr2=0x7f689c0af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.806+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 shutdown_connections 2024-04-15T15:23:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.806+0000 7f68a0b7c640 1 -- 172.21.15.12:0/523780455 wait complete. 2024-04-15T15:23:42.818 INFO:tasks.ceph:Scrubbing osd.6 2024-04-15T15:23:42.818 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:23:42.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.925+0000 7f9dc3060640 1 Processor -- start 2024-04-15T15:23:42.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.925+0000 7f9dc3060640 1 -- start start 2024-04-15T15:23:42.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.926+0000 7f9dc3060640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9dbc15e810 0x7f9dbc0d2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.926+0000 7f9dc3060640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9dbc0d4a50 0x7f9dbc0cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.926+0000 7f9dc3060640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dbc0d1f50 con 0x7f9dbc0d3320 2024-04-15T15:23:42.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.926+0000 7f9dc3060640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9dbc0d1c70 con 0x7f9dbc15e810 2024-04-15T15:23:42.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.926+0000 7f9dc3060640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9dbc0d2300 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.926+0000 7f9dc0dfc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9dbc15e810 0x7f9dbc0d2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.926+0000 7f9dc0dfc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9dbc15e810 0x7f9dbc0d2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37464/0 (socket says 172.21.15.12:37464) 2024-04-15T15:23:42.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.926+0000 7f9dc0dfc640 1 -- 172.21.15.12:0/2869305462 learned_addr learned my addr 172.21.15.12:0/2869305462 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.927+0000 7f9dc15fd640 1 --2- 172.21.15.12:0/2869305462 >> v2:172.21.15.55:3300/0 conn(0x7f9dbc0d4a50 0x7f9dbc0cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.927+0000 7f9dc0dfc640 1 -- 172.21.15.12:0/2869305462 >> v2:172.21.15.55:3300/0 conn(0x7f9dbc0d4a50 msgr2=0x7f9dbc0cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:42.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.927+0000 7f9dc0dfc640 1 --2- 172.21.15.12:0/2869305462 >> v2:172.21.15.55:3300/0 conn(0x7f9dbc0d4a50 0x7f9dbc0cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.927+0000 7f9dc0dfc640 1 -- 172.21.15.12:0/2869305462 >> v1:172.21.15.12:6789/0 conn(0x7f9dbc0d3320 legacy=0x7f9dbc0d4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.927+0000 7f9dc0dfc640 1 -- 172.21.15.12:0/2869305462 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dbc0d25e0 con 0x7f9dbc15e810 2024-04-15T15:23:42.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.927+0000 7f9dc0dfc640 1 --2- 172.21.15.12:0/2869305462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc15e810 0x7f9dbc0d2de0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0d0de0 tx=0x7f9dac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b9588d87e4bfe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dbb7fe640 1 -- 172.21.15.12:0/2869305462 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac034070 con 0x7f9dbc15e810 2024-04-15T15:23:42.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dbb7fe640 1 -- 172.21.15.12:0/2869305462 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dac030860 con 0x7f9dbc15e810 2024-04-15T15:23:42.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dbb7fe640 1 -- 172.21.15.12:0/2869305462 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac030c30 con 0x7f9dbc15e810 2024-04-15T15:23:42.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dc3060640 1 -- 172.21.15.12:0/2869305462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc15e810 msgr2=0x7f9dbc0d2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dc3060640 1 --2- 172.21.15.12:0/2869305462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc15e810 0x7f9dbc0d2de0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0d0de0 tx=0x7f9dac02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dc3060640 1 -- 172.21.15.12:0/2869305462 shutdown_connections 2024-04-15T15:23:42.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dc3060640 1 --2- 172.21.15.12:0/2869305462 >> v2:172.21.15.55:3300/0 conn(0x7f9dbc0d4a50 0x7f9dbc0cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dc3060640 1 --2- 172.21.15.12:0/2869305462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc15e810 0x7f9dbc0d2de0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dc3060640 1 -- 172.21.15.12:0/2869305462 >> 172.21.15.12:0/2869305462 conn(0x7f9dbc08e050 msgr2=0x7f9dbc09d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dc3060640 1 -- 172.21.15.12:0/2869305462 shutdown_connections 2024-04-15T15:23:42.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.928+0000 7f9dc3060640 1 -- 172.21.15.12:0/2869305462 wait complete. 2024-04-15T15:23:42.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.929+0000 7f9dc3060640 1 Processor -- start 2024-04-15T15:23:42.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.929+0000 7f9dc3060640 1 -- start start 2024-04-15T15:23:42.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc3060640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc0d4a50 0x7f9dbc0cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc3060640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9dbc15e810 0x7f9dbc1e8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc3060640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9dbc1614c0 0x7f9dbc15ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc3060640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9dbc0cf6e0 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc3060640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9dbc15ebf0 con 0x7f9dbc15e810 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc3060640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f9dbc0cf860 con 0x7f9dbc1614c0 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc0dfc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc0d4a50 0x7f9dbc0cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc0dfc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc0d4a50 0x7f9dbc0cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37474/0 (socket says 172.21.15.12:37474) 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.930+0000 7f9dc0dfc640 1 -- 172.21.15.12:0/1672503486 learned_addr learned my addr 172.21.15.12:0/1672503486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dbbfff640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9dbc15e810 0x7f9dbc1e8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc15fd640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9dbc1614c0 0x7f9dbc15ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc0dfc640 1 -- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9dbc1614c0 msgr2=0x7f9dbc15ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc0dfc640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f9dbc1614c0 0x7f9dbc15ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc0dfc640 1 -- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9dbc15e810 msgr2=0x7f9dbc1e8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc0dfc640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9dbc15e810 0x7f9dbc1e8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc0dfc640 1 -- 172.21.15.12:0/1672503486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dbc0cbe60 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dbbfff640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9dbc15e810 0x7f9dbc1e8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:42.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc0dfc640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc0d4a50 0x7f9dbc0cb760 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9dac000c00 tx=0x7f9dac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac034040 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dac039d90 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dbc0cc140 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac043610 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.931+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dbc0cc700 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.932+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9dbc1e91d0 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.933+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9dac037070 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.933+0000 7f9db97fa640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9da806c980 0x7f9da806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.933+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f9dac0ade40 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.933+0000 7f9dbbfff640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9da806c980 0x7f9da806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.933+0000 7f9db97fa640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f9da8072ec0 0x7f9da8075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.933+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 --> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9dac0067f0 con 0x7f9da8072ec0 2024-04-15T15:23:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.933+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f9dac0b0590 con 0x7f9dbc0d4a50 2024-04-15T15:23:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.933+0000 7f9dc15fd640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f9da8072ec0 0x7f9da8075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.934+0000 7f9dbbfff640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9da806c980 0x7f9da806ee40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc15c920 tx=0x7f9db0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.934+0000 7f9dc15fd640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f9da8072ec0 0x7f9da8075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.934+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 <== osd.6 v2:172.21.15.55:6816/2237244484 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9dac0067f0 con 0x7f9da8072ec0 2024-04-15T15:23:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 --> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9dbc0690f0 con 0x7f9da8072ec0 2024-04-15T15:23:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9db97fa640 1 -- 172.21.15.12:0/1672503486 <== osd.6 v2:172.21.15.55:6816/2237244484 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9dbc0690f0 con 0x7f9da8072ec0 2024-04-15T15:23:42.949 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:42.950 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:23:42.950 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:42.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f9da8072ec0 msgr2=0x7f9da8075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6816/2237244484,v1:172.21.15.55:6817/2237244484] conn(0x7f9da8072ec0 0x7f9da8075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9da806c980 msgr2=0x7f9da806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f9da806c980 0x7f9da806ee40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc15c920 tx=0x7f9db0009040 comp rx=0 tx=0).stop 2024-04-15T15:23:42.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc0d4a50 msgr2=0x7f9dbc0cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:42.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 --2- 172.21.15.12:0/1672503486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dbc0d4a50 0x7f9dbc0cb760 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9dac000c00 tx=0x7f9dac0046c0 comp rx=0 tx=0).stop 2024-04-15T15:23:42.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc15fd640 1 -- 172.21.15.12:0/1672503486 reap_dead start 2024-04-15T15:23:42.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 shutdown_connections 2024-04-15T15:23:42.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 >> 172.21.15.12:0/1672503486 conn(0x7f9dbc08e050 msgr2=0x7f9dbc162790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:42.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 shutdown_connections 2024-04-15T15:23:42.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:42.948+0000 7f9dc3060640 1 -- 172.21.15.12:0/1672503486 wait complete. 2024-04-15T15:23:42.960 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-15T15:23:43.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.065+0000 7f830fa7b640 1 Processor -- start 2024-04-15T15:23:43.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.066+0000 7f830fa7b640 1 -- start start 2024-04-15T15:23:43.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.066+0000 7f830fa7b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f830815e700 0x7f83080b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.066+0000 7f830fa7b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83080d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.066+0000 7f830fa7b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f830815eae0 0x7f83080d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.066+0000 7f830fa7b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f83080b53b0 con 0x7f830815e700 2024-04-15T15:23:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.066+0000 7f830fa7b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f83080b47a0 con 0x7f830815eae0 2024-04-15T15:23:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.066+0000 7f830fa7b640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f83080b4a80 con 0x7f83080d2360 2024-04-15T15:23:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d817640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f830815e700 0x7f83080b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d817640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f830815e700 0x7f83080b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37486/0 (socket says 172.21.15.12:37486) 2024-04-15T15:23:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d817640 1 -- 172.21.15.12:0/1204437953 learned_addr learned my addr 172.21.15.12:0/1204437953 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:43.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d817640 1 -- 172.21.15.12:0/1204437953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 msgr2=0x7f83080d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:43.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d016640 1 --2- 172.21.15.12:0/1204437953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83080d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d817640 1 --2- 172.21.15.12:0/1204437953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83080d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d817640 1 -- 172.21.15.12:0/1204437953 >> v2:172.21.15.55:3300/0 conn(0x7f830815eae0 msgr2=0x7f83080d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:43.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830e018640 1 --2- 172.21.15.12:0/1204437953 >> v2:172.21.15.55:3300/0 conn(0x7f830815eae0 0x7f83080d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d817640 1 --2- 172.21.15.12:0/1204437953 >> v2:172.21.15.55:3300/0 conn(0x7f830815eae0 0x7f83080d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.067+0000 7f830d817640 1 -- 172.21.15.12:0/1204437953 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83080b4d60 con 0x7f830815e700 2024-04-15T15:23:43.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830d817640 1 --2- 172.21.15.12:0/1204437953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080b5840 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f830808c780 tx=0x7f830402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b5983d1f658e80 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830c815640 1 -- 172.21.15.12:0/1204437953 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830402d040 con 0x7f830815e700 2024-04-15T15:23:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830c815640 1 -- 172.21.15.12:0/1204437953 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8304004230 con 0x7f830815e700 2024-04-15T15:23:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830c815640 1 -- 172.21.15.12:0/1204437953 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304003610 con 0x7f830815e700 2024-04-15T15:23:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830fa7b640 1 -- 172.21.15.12:0/1204437953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 msgr2=0x7f83080b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830fa7b640 1 --2- 172.21.15.12:0/1204437953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080b5840 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f830808c780 tx=0x7f830402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830fa7b640 1 -- 172.21.15.12:0/1204437953 shutdown_connections 2024-04-15T15:23:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830fa7b640 1 --2- 172.21.15.12:0/1204437953 >> v2:172.21.15.55:3300/0 conn(0x7f830815eae0 0x7f83080d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830fa7b640 1 --2- 172.21.15.12:0/1204437953 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83080d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830fa7b640 1 --2- 172.21.15.12:0/1204437953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080b5840 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.068+0000 7f830fa7b640 1 -- 172.21.15.12:0/1204437953 >> 172.21.15.12:0/1204437953 conn(0x7f830808bf60 msgr2=0x7f83080d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:43.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.069+0000 7f830fa7b640 1 -- 172.21.15.12:0/1204437953 shutdown_connections 2024-04-15T15:23:43.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.069+0000 7f830fa7b640 1 -- 172.21.15.12:0/1204437953 wait complete. 2024-04-15T15:23:43.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830fa7b640 1 Processor -- start 2024-04-15T15:23:43.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830fa7b640 1 -- start start 2024-04-15T15:23:43.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830fa7b640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83081f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830fa7b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830fa7b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f830815eae0 0x7f83081f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830fa7b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f83080b5d80 con 0x7f830815e700 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830fa7b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f83080b4450 con 0x7f830815eae0 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830fa7b640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f83080d1d40 con 0x7f83080d2360 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.070+0000 7f830d016640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d016640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37498/0 (socket says 172.21.15.12:37498) 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d016640 1 -- 172.21.15.12:0/849194042 learned_addr learned my addr 172.21.15.12:0/849194042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830e018640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f830815eae0 0x7f83081f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d817640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83081f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d016640 1 -- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 msgr2=0x7f83081f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d016640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83081f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d016640 1 -- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f830815eae0 msgr2=0x7f83081f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d016640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f830815eae0 0x7f83081f4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d016640 1 -- 172.21.15.12:0/849194042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83080cb650 con 0x7f830815e700 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d817640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83081f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830d016640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080cc0f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c570 tx=0x7f82f800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f82fe7fc640 1 -- 172.21.15.12:0/849194042 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8015070 con 0x7f830815e700 2024-04-15T15:23:43.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f82fe7fc640 1 -- 172.21.15.12:0/849194042 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f8011800 con 0x7f830815e700 2024-04-15T15:23:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83081f1cd0 con 0x7f830815e700 2024-04-15T15:23:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f82fe7fc640 1 -- 172.21.15.12:0/849194042 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8011bf0 con 0x7f830815e700 2024-04-15T15:23:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.071+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83081f5160 con 0x7f830815e700 2024-04-15T15:23:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.072+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83080689e0 con 0x7f830815e700 2024-04-15T15:23:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.073+0000 7f82fe7fc640 1 -- 172.21.15.12:0/849194042 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f82f8018070 con 0x7f830815e700 2024-04-15T15:23:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.073+0000 7f82fe7fc640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f82ec06c980 0x7f82ec06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.073+0000 7f82fe7fc640 1 -- 172.21.15.12:0/849194042 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f82f808b970 con 0x7f830815e700 2024-04-15T15:23:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.073+0000 7f830d817640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f82ec06c980 0x7f82ec06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.074+0000 7f830d817640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f82ec06c980 0x7f82ec06ee40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8304035000 tx=0x7f8304008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.077+0000 7f82fe7fc640 1 -- 172.21.15.12:0/849194042 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f82f808de20 con 0x7f830815e700 2024-04-15T15:23:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.265+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f83080ba500 con 0x7f82ec06c980 2024-04-15T15:23:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.265+0000 7f82fe7fc640 1 -- 172.21.15.12:0/849194042 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f83080ba500 con 0x7f82ec06c980 2024-04-15T15:23:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 6 to deep-scrub 2024-04-15T15:23:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f82ec06c980 msgr2=0x7f82ec06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f82ec06c980 0x7f82ec06ee40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8304035000 tx=0x7f8304008030 comp rx=0 tx=0).stop 2024-04-15T15:23:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 msgr2=0x7f83080cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080cc0f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c570 tx=0x7f82f800ca30 comp rx=0 tx=0).stop 2024-04-15T15:23:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 shutdown_connections 2024-04-15T15:23:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f82ec06c980 0x7f82ec06ee40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f830815eae0 0x7f83081f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f830815e700 0x7f83080cc0f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 --2- 172.21.15.12:0/849194042 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f83080d2360 0x7f83081f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.267+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 >> 172.21.15.12:0/849194042 conn(0x7f830808bf60 msgr2=0x7f83080d0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.268+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 shutdown_connections 2024-04-15T15:23:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.268+0000 7f830fa7b640 1 -- 172.21.15.12:0/849194042 wait complete. 2024-04-15T15:23:43.279 INFO:tasks.ceph:Scrubbing osd.7 2024-04-15T15:23:43.280 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:23:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.387+0000 7f45abf74640 1 Processor -- start 2024-04-15T15:23:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.387+0000 7f45abf74640 1 -- start start 2024-04-15T15:23:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45abf74640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f45a415e930 0x7f45a40ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45abf74640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45a40d16e0 0x7f45a40d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45abf74640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a415ed10 con 0x7f45a40b77b0 2024-04-15T15:23:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45abf74640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f45a40d2070 con 0x7f45a40d16e0 2024-04-15T15:23:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45abf74640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f45a40b8bc0 con 0x7f45a415e930 2024-04-15T15:23:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45aa511640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45a40d16e0 0x7f45a40d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45aa511640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f45a40d16e0 0x7f45a40d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37504/0 (socket says 172.21.15.12:37504) 2024-04-15T15:23:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45aa511640 1 -- 172.21.15.12:0/2224372402 learned_addr learned my addr 172.21.15.12:0/2224372402 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45a9d10640 1 --2- 172.21.15.12:0/2224372402 >> v2:172.21.15.55:3300/0 conn(0x7f45a415e930 0x7f45a40ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45aa511640 1 -- 172.21.15.12:0/2224372402 >> v2:172.21.15.55:3300/0 conn(0x7f45a415e930 msgr2=0x7f45a40ba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45aa511640 1 --2- 172.21.15.12:0/2224372402 >> v2:172.21.15.55:3300/0 conn(0x7f45a415e930 0x7f45a40ba260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45aa511640 1 -- 172.21.15.12:0/2224372402 >> v1:172.21.15.55:6789/0 conn(0x7f45a40b77b0 legacy=0x7f45a40d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45aa511640 1 -- 172.21.15.12:0/2224372402 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a40b7fb0 con 0x7f45a40d16e0 2024-04-15T15:23:43.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.388+0000 7f45a9d10640 1 --2- 172.21.15.12:0/2224372402 >> v2:172.21.15.55:3300/0 conn(0x7f45a415e930 0x7f45a40ba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:23:43.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.389+0000 7f45aa511640 1 --2- 172.21.15.12:0/2224372402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40d16e0 0x7f45a40d1b30 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f45a406dcd0 tx=0x7f459c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3cd97456de3b4a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.389+0000 7f45a8d0e640 1 -- 172.21.15.12:0/2224372402 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c02cbb0 con 0x7f45a40d16e0 2024-04-15T15:23:43.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.389+0000 7f45a8d0e640 1 -- 172.21.15.12:0/2224372402 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f459c02cd50 con 0x7f45a40d16e0 2024-04-15T15:23:43.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.389+0000 7f45a8d0e640 1 -- 172.21.15.12:0/2224372402 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c034610 con 0x7f45a40d16e0 2024-04-15T15:23:43.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.390+0000 7f45abf74640 1 -- 172.21.15.12:0/2224372402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40d16e0 msgr2=0x7f45a40d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.390+0000 7f45abf74640 1 --2- 172.21.15.12:0/2224372402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40d16e0 0x7f45a40d1b30 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f45a406dcd0 tx=0x7f459c02c010 comp rx=0 tx=0).stop 2024-04-15T15:23:43.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.390+0000 7f45abf74640 1 -- 172.21.15.12:0/2224372402 shutdown_connections 2024-04-15T15:23:43.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.390+0000 7f45abf74640 1 --2- 172.21.15.12:0/2224372402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40d16e0 0x7f45a40d1b30 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.390+0000 7f45abf74640 1 --2- 172.21.15.12:0/2224372402 >> v2:172.21.15.55:3300/0 conn(0x7f45a415e930 0x7f45a40ba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.390+0000 7f45abf74640 1 -- 172.21.15.12:0/2224372402 >> 172.21.15.12:0/2224372402 conn(0x7f45a4067b10 msgr2=0x7f45a4067f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:43.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.390+0000 7f45abf74640 1 -- 172.21.15.12:0/2224372402 shutdown_connections 2024-04-15T15:23:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.390+0000 7f45abf74640 1 -- 172.21.15.12:0/2224372402 wait complete. 2024-04-15T15:23:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.391+0000 7f45abf74640 1 Processor -- start 2024-04-15T15:23:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.391+0000 7f45abf74640 1 -- start start 2024-04-15T15:23:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45abf74640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40b77b0 0x7f45a4161890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45abf74640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a40d16e0 0x7f45a40ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45abf74640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45a415e930 0x7f45a41f7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45abf74640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f45a40b8850 con 0x7f45a40b77b0 2024-04-15T15:23:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45abf74640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f45a40b8570 con 0x7f45a40d16e0 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45abf74640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f45a40b7bf0 con 0x7f45a415e930 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40b77b0 0x7f45a4161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40b77b0 0x7f45a4161890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37518/0 (socket says 172.21.15.12:37518) 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 -- 172.21.15.12:0/2112564126 learned_addr learned my addr 172.21.15.12:0/2112564126 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a950f640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a40d16e0 0x7f45a40ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45aa511640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45a415e930 0x7f45a41f7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 -- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45a415e930 msgr2=0x7f45a41f7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f45a415e930 0x7f45a41f7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 -- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a40d16e0 msgr2=0x7f45a40ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a40d16e0 0x7f45a40ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 -- 172.21.15.12:0/2112564126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a41f8a10 con 0x7f45a40b77b0 2024-04-15T15:23:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45a9d10640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40b77b0 0x7f45a4161890 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f45a000e7e0 tx=0x7f45a000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a000f040 con 0x7f45a40b77b0 2024-04-15T15:23:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45a0004030 con 0x7f45a40b77b0 2024-04-15T15:23:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a41f8cf0 con 0x7f45a40b77b0 2024-04-15T15:23:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0004400 con 0x7f45a40b77b0 2024-04-15T15:23:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.392+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a41512e0 con 0x7f45a40b77b0 2024-04-15T15:23:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.393+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f45a40689e0 con 0x7f45a40b77b0 2024-04-15T15:23:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.394+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f45a000c6d0 con 0x7f45a40b77b0 2024-04-15T15:23:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.394+0000 7f459a7fc640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f458806c980 0x7f458806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.394+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f45a008b9a0 con 0x7f45a40b77b0 2024-04-15T15:23:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.394+0000 7f459a7fc640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7f4588072ec0 0x7f4588075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.394+0000 7f45a950f640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f458806c980 0x7f458806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.394+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 --> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45880759d0 con 0x7f4588072ec0 2024-04-15T15:23:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.394+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f45a00902c0 con 0x7f45a40b77b0 2024-04-15T15:23:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.395+0000 7f45aa511640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7f4588072ec0 0x7f4588075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.395+0000 7f45a950f640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f458806c980 0x7f458806ee40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4594007fd0 tx=0x7f4594009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.395+0000 7f45aa511640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7f4588072ec0 0x7f4588075300 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.396+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 <== osd.7 v2:172.21.15.55:6800/4101047654 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45880759d0 con 0x7f4588072ec0 2024-04-15T15:23:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.409+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 --> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f45a40d2570 con 0x7f4588072ec0 2024-04-15T15:23:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f459a7fc640 1 -- 172.21.15.12:0/2112564126 <== osd.7 v2:172.21.15.55:6800/4101047654 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f45a40d2570 con 0x7f4588072ec0 2024-04-15T15:23:43.413 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-15T15:23:43.413 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:23:43.413 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-15T15:23:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7f4588072ec0 msgr2=0x7f4588075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6800/4101047654,v1:172.21.15.55:6801/4101047654] conn(0x7f4588072ec0 0x7f4588075300 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f458806c980 msgr2=0x7f458806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f458806c980 0x7f458806ee40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4594007fd0 tx=0x7f4594009040 comp rx=0 tx=0).stop 2024-04-15T15:23:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40b77b0 msgr2=0x7f45a4161890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 --2- 172.21.15.12:0/2112564126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a40b77b0 0x7f45a4161890 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f45a000e7e0 tx=0x7f45a000eca0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45aa511640 1 -- 172.21.15.12:0/2112564126 reap_dead start 2024-04-15T15:23:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 shutdown_connections 2024-04-15T15:23:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 >> 172.21.15.12:0/2112564126 conn(0x7f45a4067b10 msgr2=0x7f45a40d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 shutdown_connections 2024-04-15T15:23:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.410+0000 7f45abf74640 1 -- 172.21.15.12:0/2112564126 wait complete. 2024-04-15T15:23:43.422 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-15T15:23:43.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.526+0000 7f78cf24a640 1 Processor -- start 2024-04-15T15:23:43.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.526+0000 7f78cf24a640 1 -- start start 2024-04-15T15:23:43.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78cf24a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f78c809b880 0x7f78c8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78cf24a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c80d05e0 con 0x7f78c8070940 2024-04-15T15:23:43.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78c7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f78c809b880 0x7f78c8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78ccfe6640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f78c8070940 0x7f78c8160310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54986/0 (socket says 172.21.15.12:54986) 2024-04-15T15:23:43.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78cf24a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c8071260 con 0x7f78c80d0da0 2024-04-15T15:23:43.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78ccfe6640 1 -- 172.21.15.12:0/2017202223 learned_addr learned my addr 172.21.15.12:0/2017202223 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:23:43.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78cf24a640 1 -- 172.21.15.12:0/2017202223 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f78c809bce0 con 0x7f78c809b880 2024-04-15T15:23:43.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78c7fff640 1 -- 172.21.15.12:0/2017202223 >> v1:172.21.15.55:6789/0 conn(0x7f78c80d0da0 legacy=0x7f78c80d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78c7fff640 1 -- 172.21.15.12:0/2017202223 >> v1:172.21.15.12:6789/0 conn(0x7f78c8070940 legacy=0x7f78c8160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.527+0000 7f78c7fff640 1 -- 172.21.15.12:0/2017202223 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c815f720 con 0x7f78c809b880 2024-04-15T15:23:43.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.528+0000 7f78c7fff640 1 --2- 172.21.15.12:0/2017202223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c809b880 0x7f78c8070d20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0097a0 tx=0x7f78bc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604db09597d336be server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.528+0000 7f78c77fe640 1 -- 172.21.15.12:0/2017202223 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc034070 con 0x7f78c809b880 2024-04-15T15:23:43.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.528+0000 7f78c77fe640 1 -- 172.21.15.12:0/2017202223 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78bc030610 con 0x7f78c809b880 2024-04-15T15:23:43.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.528+0000 7f78c77fe640 1 -- 172.21.15.12:0/2017202223 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc0309e0 con 0x7f78c809b880 2024-04-15T15:23:43.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.529+0000 7f78cf24a640 1 -- 172.21.15.12:0/2017202223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c809b880 msgr2=0x7f78c8070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.529+0000 7f78cf24a640 1 --2- 172.21.15.12:0/2017202223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c809b880 0x7f78c8070d20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0097a0 tx=0x7f78bc02b960 comp rx=0 tx=0).stop 2024-04-15T15:23:43.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.529+0000 7f78cf24a640 1 -- 172.21.15.12:0/2017202223 shutdown_connections 2024-04-15T15:23:43.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.529+0000 7f78cf24a640 1 --2- 172.21.15.12:0/2017202223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c809b880 0x7f78c8070d20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.529+0000 7f78cf24a640 1 -- 172.21.15.12:0/2017202223 >> 172.21.15.12:0/2017202223 conn(0x7f78c80b3f40 msgr2=0x7f78c80b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:43.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.529+0000 7f78cf24a640 1 -- 172.21.15.12:0/2017202223 shutdown_connections 2024-04-15T15:23:43.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.529+0000 7f78cf24a640 1 -- 172.21.15.12:0/2017202223 wait complete. 2024-04-15T15:23:43.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.530+0000 7f78cf24a640 1 Processor -- start 2024-04-15T15:23:43.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.530+0000 7f78cf24a640 1 -- start start 2024-04-15T15:23:43.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cf24a640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78c8070940 0x7f78c80c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cf24a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78c809b880 0x7f78c815e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cf24a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c80d0da0 0x7f78c815e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cf24a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78c80d0880 con 0x7f78c80d0da0 2024-04-15T15:23:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cf24a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f78c815fce0 con 0x7f78c809b880 2024-04-15T15:23:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cf24a640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f78c80d18e0 con 0x7f78c8070940 2024-04-15T15:23:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cd7e7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c80d0da0 0x7f78c815e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cd7e7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c80d0da0 0x7f78c815e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37536/0 (socket says 172.21.15.12:37536) 2024-04-15T15:23:43.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cd7e7640 1 -- 172.21.15.12:0/806672815 learned_addr learned my addr 172.21.15.12:0/806672815 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78c7fff640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78c809b880 0x7f78c815e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78ccfe6640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78c8070940 0x7f78c80c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cd7e7640 1 -- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78c8070940 msgr2=0x7f78c80c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cd7e7640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78c8070940 0x7f78c80c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cd7e7640 1 -- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78c809b880 msgr2=0x7f78c815e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cd7e7640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78c809b880 0x7f78c815e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.531+0000 7f78cd7e7640 1 -- 172.21.15.12:0/806672815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c80b53d0 con 0x7f78c80d0da0 2024-04-15T15:23:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.532+0000 7f78cd7e7640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c80d0da0 0x7f78c815e6d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f78c000c7a0 tx=0x7f78c000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.532+0000 7f78c57fa640 1 -- 172.21.15.12:0/806672815 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0003b70 con 0x7f78c80d0da0 2024-04-15T15:23:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.532+0000 7f78c57fa640 1 -- 172.21.15.12:0/806672815 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78c0003d30 con 0x7f78c80d0da0 2024-04-15T15:23:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.532+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c80b56b0 con 0x7f78c80d0da0 2024-04-15T15:23:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.532+0000 7f78c57fa640 1 -- 172.21.15.12:0/806672815 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c000aa90 con 0x7f78c80d0da0 2024-04-15T15:23:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.532+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c80b5c70 con 0x7f78c80d0da0 2024-04-15T15:23:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.533+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c81f4c00 con 0x7f78c80d0da0 2024-04-15T15:23:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.533+0000 7f78c57fa640 1 -- 172.21.15.12:0/806672815 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f78c000ac30 con 0x7f78c80d0da0 2024-04-15T15:23:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.534+0000 7f78c57fa640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78b406c980 0x7f78b406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.534+0000 7f78c57fa640 1 -- 172.21.15.12:0/806672815 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f78c008b980 con 0x7f78c80d0da0 2024-04-15T15:23:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.534+0000 7f78ccfe6640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78b406c980 0x7f78b406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.535+0000 7f78ccfe6640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78b406c980 0x7f78b406ee40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f78b8005880 tx=0x7f78b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.538+0000 7f78c57fa640 1 -- 172.21.15.12:0/806672815 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78c0059f70 con 0x7f78c80d0da0 2024-04-15T15:23:43.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.721+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f78c80689e0 con 0x7f78b406c980 2024-04-15T15:23:43.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.722+0000 7f78c57fa640 1 -- 172.21.15.12:0/806672815 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f78c80689e0 con 0x7f78b406c980 2024-04-15T15:23:43.723 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 7 to deep-scrub 2024-04-15T15:23:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78b406c980 msgr2=0x7f78b406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78b406c980 0x7f78b406ee40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f78b8005880 tx=0x7f78b8009040 comp rx=0 tx=0).stop 2024-04-15T15:23:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c80d0da0 msgr2=0x7f78c815e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c80d0da0 0x7f78c815e6d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f78c000c7a0 tx=0x7f78c000cc60 comp rx=0 tx=0).stop 2024-04-15T15:23:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 shutdown_connections 2024-04-15T15:23:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78c80d0da0 0x7f78c815e6d0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f78b406c980 0x7f78b406ee40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78c809b880 0x7f78c815e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 --2- 172.21.15.12:0/806672815 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f78c8070940 0x7f78c80c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 >> 172.21.15.12:0/806672815 conn(0x7f78c80b3f40 msgr2=0x7f78c815c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 shutdown_connections 2024-04-15T15:23:43.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.724+0000 7f78cf24a640 1 -- 172.21.15.12:0/806672815 wait complete. 2024-04-15T15:23:43.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:23:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.839+0000 7f0aa49c3640 1 Processor -- start 2024-04-15T15:23:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.839+0000 7f0aa49c3640 1 -- start start 2024-04-15T15:23:43.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.839+0000 7f0aa49c3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0aa00d2360 0x7f0aa00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.839+0000 7f0aa49c3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa015eae0 0x7f0aa00d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.839+0000 7f0aa49c3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aa00b47a0 con 0x7f0aa015e700 2024-04-15T15:23:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.839+0000 7f0aa49c3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0aa00b53b0 con 0x7f0aa00d2360 2024-04-15T15:23:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.839+0000 7f0aa49c3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa00b4a80 con 0x7f0aa015eae0 2024-04-15T15:23:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a97fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0aa00d2360 0x7f0aa00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a9ed9d640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa015eae0 0x7f0aa00d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a97fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0aa00d2360 0x7f0aa00d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.12:33192/0 (socket says 172.21.15.12:33192) 2024-04-15T15:23:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a97fff640 1 -- 172.21.15.12:0/2016717077 learned_addr learned my addr 172.21.15.12:0/2016717077 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a9e59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0aa015e700 0x7f0aa00b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.12:33190/0 (socket says 172.21.15.12:33190) 2024-04-15T15:23:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a97fff640 1 -- 172.21.15.12:0/2016717077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa015eae0 msgr2=0x7f0aa00d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:23:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a97fff640 1 --2- 172.21.15.12:0/2016717077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa015eae0 0x7f0aa00d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a97fff640 1 -- 172.21.15.12:0/2016717077 >> v1:172.21.15.55:6789/0 conn(0x7f0aa015e700 legacy=0x7f0aa00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a97fff640 1 -- 172.21.15.12:0/2016717077 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa00b4d60 con 0x7f0aa00d2360 2024-04-15T15:23:43.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.840+0000 7f0a9ed9d640 1 --2- 172.21.15.12:0/2016717077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa015eae0 0x7f0aa00d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:23:43.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.841+0000 7f0a97fff640 1 --2- 172.21.15.12:0/2016717077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa00d2360 0x7f0aa00d2b60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009870 tx=0x7f0a8802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89f9c8d7b3fcaddd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.841+0000 7f0a9dd9b640 1 -- 172.21.15.12:0/2016717077 <== mon.1 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88034070 con 0x7f0aa00d2360 2024-04-15T15:23:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.841+0000 7f0a9dd9b640 1 -- 172.21.15.12:0/2016717077 <== mon.1 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a88030840 con 0x7f0aa00d2360 2024-04-15T15:23:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.841+0000 7f0a9dd9b640 1 -- 172.21.15.12:0/2016717077 <== mon.1 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88030c10 con 0x7f0aa00d2360 2024-04-15T15:23:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.842+0000 7f0aa49c3640 1 -- 172.21.15.12:0/2016717077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa00d2360 msgr2=0x7f0aa00d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.842+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/2016717077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa00d2360 0x7f0aa00d2b60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009870 tx=0x7f0a8802b980 comp rx=0 tx=0).stop 2024-04-15T15:23:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.842+0000 7f0aa49c3640 1 -- 172.21.15.12:0/2016717077 shutdown_connections 2024-04-15T15:23:43.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.842+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/2016717077 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa015eae0 0x7f0aa00d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.842+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/2016717077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa00d2360 0x7f0aa00d2b60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.842+0000 7f0aa49c3640 1 -- 172.21.15.12:0/2016717077 >> 172.21.15.12:0/2016717077 conn(0x7f0aa008bf60 msgr2=0x7f0aa00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.842+0000 7f0aa49c3640 1 -- 172.21.15.12:0/2016717077 shutdown_connections 2024-04-15T15:23:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.842+0000 7f0aa49c3640 1 -- 172.21.15.12:0/2016717077 wait complete. 2024-04-15T15:23:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.843+0000 7f0aa49c3640 1 Processor -- start 2024-04-15T15:23:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.843+0000 7f0aa49c3640 1 -- start start 2024-04-15T15:23:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0aa49c3640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa00d2360 0x7f0aa00b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0aa49c3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa015e700 0x7f0aa015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0aa49c3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa015eae0 0x7f0aa015aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0aa49c3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa00d1d40 con 0x7f0aa015e700 2024-04-15T15:23:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0aa49c3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa00b4450 con 0x7f0aa015eae0 2024-04-15T15:23:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0aa49c3640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa00d1ec0 con 0x7f0aa00d2360 2024-04-15T15:23:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a97fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa015e700 0x7f0aa015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a97fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa015e700 0x7f0aa015a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37538/0 (socket says 172.21.15.12:37538) 2024-04-15T15:23:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a97fff640 1 -- 172.21.15.12:0/918183661 learned_addr learned my addr 172.21.15.12:0/918183661 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:23:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a9ed9d640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa015eae0 0x7f0aa015aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a9e59c640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa00d2360 0x7f0aa00b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a97fff640 1 -- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa00d2360 msgr2=0x7f0aa00b73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a97fff640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa00d2360 0x7f0aa00b73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a97fff640 1 -- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa015eae0 msgr2=0x7f0aa015aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:43.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a97fff640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa015eae0 0x7f0aa015aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:43.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.844+0000 7f0a97fff640 1 -- 172.21.15.12:0/918183661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa00cd5b0 con 0x7f0aa015e700 2024-04-15T15:23:43.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.845+0000 7f0a97fff640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa015e700 0x7f0aa015a530 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0a8802b9c0 tx=0x7f0a88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.845+0000 7f0a96ffd640 1 -- 172.21.15.12:0/918183661 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88048070 con 0x7f0aa015e700 2024-04-15T15:23:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.845+0000 7f0a96ffd640 1 -- 172.21.15.12:0/918183661 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a88030640 con 0x7f0aa015e700 2024-04-15T15:23:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.845+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa00cd890 con 0x7f0aa015e700 2024-04-15T15:23:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.845+0000 7f0a96ffd640 1 -- 172.21.15.12:0/918183661 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88034040 con 0x7f0aa015e700 2024-04-15T15:23:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.845+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa00cde50 con 0x7f0aa015e700 2024-04-15T15:23:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.846+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aa00689e0 con 0x7f0aa015e700 2024-04-15T15:23:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.846+0000 7f0a96ffd640 1 -- 172.21.15.12:0/918183661 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0a8804b070 con 0x7f0aa015e700 2024-04-15T15:23:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.847+0000 7f0a96ffd640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c980 0x7f0a8006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:23:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.847+0000 7f0a96ffd640 1 -- 172.21.15.12:0/918183661 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f0a880ad8e0 con 0x7f0aa015e700 2024-04-15T15:23:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.847+0000 7f0a9e59c640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c980 0x7f0a8006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:23:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.847+0000 7f0a9e59c640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c980 0x7f0a8006ee40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0a90000fb0 tx=0x7f0a90007e70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:23:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:43.851+0000 7f0a96ffd640 1 -- 172.21.15.12:0/918183661 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a8807b6e0 con 0x7f0aa015e700 2024-04-15T15:23:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.034+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0aa00b4210 con 0x7f0a8006c980 2024-04-15T15:23:44.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.035+0000 7f0a96ffd640 1 -- 172.21.15.12:0/918183661 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13423 (secure 0 0 0) 0x7f0aa00b4210 con 0x7f0a8006c980 2024-04-15T15:23:44.037 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:23:44.037 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:23:44.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c980 msgr2=0x7f0a8006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:44.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c980 0x7f0a8006ee40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0a90000fb0 tx=0x7f0a90007e70 comp rx=0 tx=0).stop 2024-04-15T15:23:44.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa015e700 msgr2=0x7f0aa015a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:23:44.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa015e700 0x7f0aa015a530 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0a8802b9c0 tx=0x7f0a88004730 comp rx=0 tx=0).stop 2024-04-15T15:23:44.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 shutdown_connections 2024-04-15T15:23:44.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f0a8006c980 0x7f0a8006ee40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:44.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0aa015eae0 0x7f0aa015aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:44.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa015e700 0x7f0aa015a530 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:44.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 --2- 172.21.15.12:0/918183661 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f0aa00d2360 0x7f0aa00b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:23:44.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 >> 172.21.15.12:0/918183661 conn(0x7f0aa008bf60 msgr2=0x7f0aa00d0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:23:44.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 shutdown_connections 2024-04-15T15:23:44.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:23:44.037+0000 7f0aa49c3640 1 -- 172.21.15.12:0/918183661 wait complete. 2024-04-15T15:23:44.049 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-04-15T15:23:42.625152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238164,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736556,"statfs":{"total":773094113280,"available":772850233344,"internally_reserved":0,"allocated":5447680,"data_stored":3047840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-133,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.910807"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":216,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-15T15:23:34.855100+0000","last_change":"2024-04-15T15:21:00.585832+0000","last_active":"2024-04-15T15:23:34.855100+0000","last_peered":"2024-04-15T15:23:34.855100+0000","last_clean":"2024-04-15T15:23:34.855100+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:23:34.855100+0000","last_undegraded":"2024-04-15T15:23:34.855100+0000","last_fullsized":"2024-04-15T15:23:34.855100+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:20:59.514865+0000","last_clean_scrub_stamp":"2024-04-15T15:20:59.514865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:27:51.690464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1126400,"data_stored":823756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27544,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1126400,"data_stored":823756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:23:44.052 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-15T15:20:59.514865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=15, tm_hour=15, tm_min=20, tm_sec=59, tm_wday=0, tm_yday=106, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=15, tm_hour=15, tm_min=23, tm_sec=38, tm_wday=0, tm_yday=106, tm_isdst=0) 2024-04-15T15:23:44.052 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-15T15:24:01.824 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:24:01.827 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:24:04.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:24:04.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.168+0000 7f74ab3af640 1 Processor -- start 2024-04-15T15:24:04.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.168+0000 7f74ab3af640 1 -- start start 2024-04-15T15:24:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74ab3af640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f74a40af340 0x7f74a40b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74ab3af640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74a40b1620 0x7f74a40b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74ab3af640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a40af920 con 0x7f74a415ecb0 2024-04-15T15:24:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74ab3af640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f74a415f090 con 0x7f74a40af340 2024-04-15T15:24:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74ab3af640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f74a40cdf80 con 0x7f74a40b1620 2024-04-15T15:24:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74a894a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f74a40af340 0x7f74a40b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74a894a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f74a40af340 0x7f74a40b4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59076/0 (socket says 172.21.15.12:59076) 2024-04-15T15:24:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74a894a640 1 -- 172.21.15.12:0/3261872455 learned_addr learned my addr 172.21.15.12:0/3261872455 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:24:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74a994c640 1 --2- 172.21.15.12:0/3261872455 >> v2:172.21.15.55:3300/0 conn(0x7f74a40b1620 0x7f74a40b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74a894a640 1 -- 172.21.15.12:0/3261872455 >> v2:172.21.15.55:3300/0 conn(0x7f74a40b1620 msgr2=0x7f74a40b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:24:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74a894a640 1 --2- 172.21.15.12:0/3261872455 >> v2:172.21.15.55:3300/0 conn(0x7f74a40b1620 0x7f74a40b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74a894a640 1 -- 172.21.15.12:0/3261872455 >> v1:172.21.15.12:6789/0 conn(0x7f74a415ecb0 legacy=0x7f74a40b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.169+0000 7f74a894a640 1 -- 172.21.15.12:0/3261872455 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a40afc00 con 0x7f74a40af340 2024-04-15T15:24:04.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.170+0000 7f74a894a640 1 --2- 172.21.15.12:0/3261872455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a40b4350 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f74a40cf770 tx=0x7f749402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f68e9d8c401de6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.170+0000 7f749bfff640 1 -- 172.21.15.12:0/3261872455 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749402f850 con 0x7f74a40af340 2024-04-15T15:24:04.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.170+0000 7f749bfff640 1 -- 172.21.15.12:0/3261872455 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f749402f9f0 con 0x7f74a40af340 2024-04-15T15:24:04.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.170+0000 7f749bfff640 1 -- 172.21.15.12:0/3261872455 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749402fdc0 con 0x7f74a40af340 2024-04-15T15:24:04.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.171+0000 7f74ab3af640 1 -- 172.21.15.12:0/3261872455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 msgr2=0x7f74a40b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.171+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3261872455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a40b4350 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f74a40cf770 tx=0x7f749402ba30 comp rx=0 tx=0).stop 2024-04-15T15:24:04.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.171+0000 7f74ab3af640 1 -- 172.21.15.12:0/3261872455 shutdown_connections 2024-04-15T15:24:04.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.171+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3261872455 >> v2:172.21.15.55:3300/0 conn(0x7f74a40b1620 0x7f74a40b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.171+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3261872455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a40b4350 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.171+0000 7f74ab3af640 1 -- 172.21.15.12:0/3261872455 >> 172.21.15.12:0/3261872455 conn(0x7f74a40b6870 msgr2=0x7f74a40b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:24:04.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.171+0000 7f74ab3af640 1 -- 172.21.15.12:0/3261872455 shutdown_connections 2024-04-15T15:24:04.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.171+0000 7f74ab3af640 1 -- 172.21.15.12:0/3261872455 wait complete. 2024-04-15T15:24:04.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.172+0000 7f74ab3af640 1 Processor -- start 2024-04-15T15:24:04.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.172+0000 7f74ab3af640 1 -- start start 2024-04-15T15:24:04.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74ab3af640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a4157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74ab3af640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a40b1620 0x7f74a40cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74ab3af640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a415ecb0 0x7f74a40cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74ab3af640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f74a40ce410 con 0x7f74a40af340 2024-04-15T15:24:04.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74ab3af640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f74a40b4890 con 0x7f74a40b1620 2024-04-15T15:24:04.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74ab3af640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f74a40ce590 con 0x7f74a415ecb0 2024-04-15T15:24:04.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a914b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a4157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a914b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a4157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59082/0 (socket says 172.21.15.12:59082) 2024-04-15T15:24:04.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a914b640 1 -- 172.21.15.12:0/3542685958 learned_addr learned my addr 172.21.15.12:0/3542685958 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:24:04.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a894a640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a40b1620 0x7f74a40cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a994c640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a415ecb0 0x7f74a40cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a914b640 1 -- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a415ecb0 msgr2=0x7f74a40cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a914b640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a415ecb0 0x7f74a40cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a914b640 1 -- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a40b1620 msgr2=0x7f74a40cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.173+0000 7f74a914b640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a40b1620 0x7f74a40cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.174+0000 7f74a914b640 1 -- 172.21.15.12:0/3542685958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a4160f50 con 0x7f74a40af340 2024-04-15T15:24:04.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.174+0000 7f74a914b640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a4157970 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c3c0 tx=0x7f74a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.174+0000 7f7499ffb640 1 -- 172.21.15.12:0/3542685958 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a00052d0 con 0x7f74a40af340 2024-04-15T15:24:04.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.174+0000 7f7499ffb640 1 -- 172.21.15.12:0/3542685958 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a0005490 con 0x7f74a40af340 2024-04-15T15:24:04.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.174+0000 7f7499ffb640 1 -- 172.21.15.12:0/3542685958 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0007260 con 0x7f74a40af340 2024-04-15T15:24:04.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.174+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a415feb0 con 0x7f74a40af340 2024-04-15T15:24:04.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.174+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a4160470 con 0x7f74a40af340 2024-04-15T15:24:04.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.175+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a40689e0 con 0x7f74a40af340 2024-04-15T15:24:04.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.176+0000 7f7499ffb640 1 -- 172.21.15.12:0/3542685958 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f74a0007400 con 0x7f74a40af340 2024-04-15T15:24:04.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.176+0000 7f7499ffb640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748806c980 0x7f748806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.176+0000 7f7499ffb640 1 -- 172.21.15.12:0/3542685958 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f74a008aea0 con 0x7f74a40af340 2024-04-15T15:24:04.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.177+0000 7f74a894a640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748806c980 0x7f748806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.177+0000 7f74a894a640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748806c980 0x7f748806ee40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7494036000 tx=0x7f7494008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.181+0000 7f7499ffb640 1 -- 172.21.15.12:0/3542685958 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74a0058db0 con 0x7f74a40af340 2024-04-15T15:24:04.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.364+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 --> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f74a406c7c0 con 0x7f748806c980 2024-04-15T15:24:04.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.365+0000 7f7499ffb640 1 -- 172.21.15.12:0/3542685958 <== mgr.4106 v2:172.21.15.55:6832/746784251 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7f74a406c7c0 con 0x7f748806c980 2024-04-15T15:24:04.367 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:24:04.367 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-15T15:24:04.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748806c980 msgr2=0x7f748806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748806c980 0x7f748806ee40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7494036000 tx=0x7f7494008030 comp rx=0 tx=0).stop 2024-04-15T15:24:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 msgr2=0x7f74a4157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a4157970 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c3c0 tx=0x7f74a000c880 comp rx=0 tx=0).stop 2024-04-15T15:24:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 shutdown_connections 2024-04-15T15:24:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f74a415ecb0 0x7f74a40cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f748806c980 0x7f748806ee40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74a40b1620 0x7f74a40cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 --2- 172.21.15.12:0/3542685958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74a40af340 0x7f74a4157970 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 >> 172.21.15.12:0/3542685958 conn(0x7f74a40b6870 msgr2=0x7f74a40b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:24:04.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 shutdown_connections 2024-04-15T15:24:04.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.367+0000 7f74ab3af640 1 -- 172.21.15.12:0/3542685958 wait complete. 2024-04-15T15:24:04.379 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-04-15T15:24:02.629059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238148,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736572,"statfs":{"total":773094113280,"available":772850249728,"internally_reserved":0,"allocated":5447680,"data_stored":3047840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002517"},"pg_stats":[{"pgid":"1.0","version":"45'119","reported_seq":227,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-15T15:23:44.215924+0000","last_change":"2024-04-15T15:23:44.215960+0000","last_active":"2024-04-15T15:23:44.215924+0000","last_peered":"2024-04-15T15:23:44.215924+0000","last_clean":"2024-04-15T15:23:44.215924+0000","last_became_active":"2024-04-15T15:21:00.584783+0000","last_became_peered":"2024-04-15T15:21:00.584783+0000","last_unstale":"2024-04-15T15:23:44.215924+0000","last_undegraded":"2024-04-15T15:23:44.215924+0000","last_fullsized":"2024-04-15T15:23:44.215924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'119","last_scrub_stamp":"2024-04-15T15:23:44.215831+0000","last_deep_scrub":"45'119","last_deep_scrub_stamp":"2024-04-15T15:23:44.215831+0000","last_clean_scrub_stamp":"2024-04-15T15:23:44.215831+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:26.456288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1126400,"data_stored":823756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27544,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":233388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343724,"statfs":{"total":96636764160,"available":96607973376,"internally_reserved":0,"allocated":1126400,"data_stored":823756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:24:04.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-15T15:24:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.484+0000 7f1064688640 1 Processor -- start 2024-04-15T15:24:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.484+0000 7f1064688640 1 -- start start 2024-04-15T15:24:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1064688640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f105c0d0690 0x7f105c0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1064688640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c0d0fb0 con 0x7f105c0ae280 2024-04-15T15:24:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1064688640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c15cb50 con 0x7f105c0b3160 2024-04-15T15:24:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1064688640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f105c15bda0 con 0x7f105c0d0690 2024-04-15T15:24:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1062424640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f105c0d0690 0x7f105c0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1062424640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f105c0d0690 0x7f105c0d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59088/0 (socket says 172.21.15.12:59088) 2024-04-15T15:24:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1062424640 1 -- 172.21.15.12:0/3193663167 learned_addr learned my addr 172.21.15.12:0/3193663167 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:24:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1062424640 1 -- 172.21.15.12:0/3193663167 >> v1:172.21.15.55:6789/0 conn(0x7f105c0b3160 legacy=0x7f105c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:24:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1062424640 1 -- 172.21.15.12:0/3193663167 >> v1:172.21.15.12:6789/0 conn(0x7f105c0ae280 legacy=0x7f105c0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1062424640 1 -- 172.21.15.12:0/3193663167 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c15bfd0 con 0x7f105c0d0690 2024-04-15T15:24:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.485+0000 7f1061422640 1 -- 172.21.15.12:0/3193663167 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2373901285 0 0) 0x7f105c0d0fb0 con 0x7f105c0ae280 2024-04-15T15:24:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.486+0000 7f1062424640 1 --2- 172.21.15.12:0/3193663167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0d0690 0x7f105c0d0a70 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1058009e50 tx=0x7f105802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621910b52ecd2030 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.486+0000 7f1061422640 1 -- 172.21.15.12:0/3193663167 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105802cbb0 con 0x7f105c0d0690 2024-04-15T15:24:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.486+0000 7f1061422640 1 -- 172.21.15.12:0/3193663167 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f105802cd50 con 0x7f105c0d0690 2024-04-15T15:24:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.486+0000 7f1061422640 1 -- 172.21.15.12:0/3193663167 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058034610 con 0x7f105c0d0690 2024-04-15T15:24:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.486+0000 7f1064688640 1 -- 172.21.15.12:0/3193663167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0d0690 msgr2=0x7f105c0d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.486+0000 7f1064688640 1 --2- 172.21.15.12:0/3193663167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0d0690 0x7f105c0d0a70 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1058009e50 tx=0x7f105802c010 comp rx=0 tx=0).stop 2024-04-15T15:24:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.487+0000 7f1064688640 1 -- 172.21.15.12:0/3193663167 shutdown_connections 2024-04-15T15:24:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.487+0000 7f1064688640 1 --2- 172.21.15.12:0/3193663167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0d0690 0x7f105c0d0a70 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.487+0000 7f1064688640 1 -- 172.21.15.12:0/3193663167 >> 172.21.15.12:0/3193663167 conn(0x7f105c06be00 msgr2=0x7f105c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:24:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.487+0000 7f1064688640 1 -- 172.21.15.12:0/3193663167 shutdown_connections 2024-04-15T15:24:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.487+0000 7f1064688640 1 -- 172.21.15.12:0/3193663167 wait complete. 2024-04-15T15:24:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.488+0000 7f1064688640 1 Processor -- start 2024-04-15T15:24:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.488+0000 7f1064688640 1 -- start start 2024-04-15T15:24:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.488+0000 7f1064688640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f105c0ae280 0x7f105c156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.488+0000 7f1064688640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0b3160 0x7f105c1f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.488+0000 7f1064688640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f105c0d0690 0x7f105c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.488+0000 7f1064688640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f105c15c870 con 0x7f105c0b3160 2024-04-15T15:24:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.488+0000 7f1064688640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f105c15c590 con 0x7f105c0ae280 2024-04-15T15:24:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.488+0000 7f1064688640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7f105c0b4fb0 con 0x7f105c0d0690 2024-04-15T15:24:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0b3160 0x7f105c1f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0b3160 0x7f105c1f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59104/0 (socket says 172.21.15.12:59104) 2024-04-15T15:24:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 -- 172.21.15.12:0/1238067904 learned_addr learned my addr 172.21.15.12:0/1238067904 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:24:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 -- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f105c0d0690 msgr2=0x7f105c1598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f105c0d0690 0x7f105c1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 -- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f105c0ae280 msgr2=0x7f105c156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f105c0ae280 0x7f105c156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 -- 172.21.15.12:0/1238067904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c156f60 con 0x7f105c0b3160 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1061c23640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0b3160 0x7f105c1f4fd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f104c00c400 tx=0x7f104c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1052ffd640 1 -- 172.21.15.12:0/1238067904 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c017070 con 0x7f105c0b3160 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1052ffd640 1 -- 172.21.15.12:0/1238067904 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c0052d0 con 0x7f105c0b3160 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105c1f5920 con 0x7f105c0b3160 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1052ffd640 1 -- 172.21.15.12:0/1238067904 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c003830 con 0x7f105c0b3160 2024-04-15T15:24:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.489+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f105c1f5ee0 con 0x7f105c0b3160 2024-04-15T15:24:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.490+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f105c0689e0 con 0x7f105c0b3160 2024-04-15T15:24:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.491+0000 7f1052ffd640 1 -- 172.21.15.12:0/1238067904 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f104c01a070 con 0x7f105c0b3160 2024-04-15T15:24:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.491+0000 7f1052ffd640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f104006c980 0x7f104006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.491+0000 7f1052ffd640 1 -- 172.21.15.12:0/1238067904 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f104c08acf0 con 0x7f105c0b3160 2024-04-15T15:24:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.491+0000 7f1062424640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f104006c980 0x7f104006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.492+0000 7f1062424640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f104006c980 0x7f104006ee40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1058035000 tx=0x7f105802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.495+0000 7f1052ffd640 1 -- 172.21.15.12:0/1238067904 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f104c0592e0 con 0x7f105c0b3160 2024-04-15T15:24:04.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.682+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f105c0739e0 con 0x7f105c0b3160 2024-04-15T15:24:04.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.686+0000 7f1052ffd640 1 -- 172.21.15.12:0/1238067904 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v16)=0 v16) ==== 123+0+0 (secure 0 0 0) 0x7f104c058be0 con 0x7f105c0b3160 2024-04-15T15:24:04.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.686+0000 7f1052ffd640 1 -- 172.21.15.12:0/1238067904 <== mon.0 v2:172.21.15.12:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f104c005480 con 0x7f105c0b3160 2024-04-15T15:24:04.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f104006c980 msgr2=0x7f104006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f104006c980 0x7f104006ee40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1058035000 tx=0x7f105802e000 comp rx=0 tx=0).stop 2024-04-15T15:24:04.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0b3160 msgr2=0x7f105c1f4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0b3160 0x7f105c1f4fd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f104c00c400 tx=0x7f104c00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 shutdown_connections 2024-04-15T15:24:04.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7f105c0d0690 0x7f105c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c0b3160 0x7f105c1f4fd0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7f104006c980 0x7f104006ee40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.688+0000 7f1064688640 1 --2- 172.21.15.12:0/1238067904 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f105c0ae280 0x7f105c156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.689+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 >> 172.21.15.12:0/1238067904 conn(0x7f105c06be00 msgr2=0x7f105c0b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:24:04.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.689+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 shutdown_connections 2024-04-15T15:24:04.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.689+0000 7f1064688640 1 -- 172.21.15.12:0/1238067904 wait complete. 2024-04-15T15:24:04.701 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:24:04.701 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:24:04.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.804+0000 7fe637d40640 1 Processor -- start 2024-04-15T15:24:04.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.805+0000 7fe637d40640 1 -- start start 2024-04-15T15:24:04.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.805+0000 7fe637d40640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 0x7fe6300b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.805+0000 7fe637d40640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe6300d3270 0x7fe6300cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.805+0000 7fe637d40640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6300b47a0 con 0x7fe6300d2360 2024-04-15T15:24:04.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.805+0000 7fe637d40640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe6300b53b0 con 0x7fe6300d3270 2024-04-15T15:24:04.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.805+0000 7fe637d40640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe6300b4a80 con 0x7fe63015e700 2024-04-15T15:24:04.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.805+0000 7fe6352db640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe6300d2360 0x7fe6300d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60402/0 (socket says 172.21.15.12:60402) 2024-04-15T15:24:04.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.805+0000 7fe6352db640 1 -- 172.21.15.12:0/4187008293 learned_addr learned my addr 172.21.15.12:0/4187008293 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-04-15T15:24:04.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe635adc640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 0x7fe6300b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe6362dd640 1 --2- 172.21.15.12:0/4187008293 >> v2:172.21.15.55:3300/0 conn(0x7fe6300d3270 0x7fe6300cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224217421 0 0) 0x7fe6300b47a0 con 0x7fe6300d2360 2024-04-15T15:24:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe618003520 con 0x7fe6300d2360 2024-04-15T15:24:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4209419533 0 0) 0x7fe618003520 con 0x7fe6300d2360 2024-04-15T15:24:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 msgr2=0x7fe6300b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:24:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 0x7fe6300b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 >> v2:172.21.15.55:3300/0 conn(0x7fe6300d3270 msgr2=0x7fe6300cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:24:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 --2- 172.21.15.12:0/4187008293 >> v2:172.21.15.55:3300/0 conn(0x7fe6300d3270 0x7fe6300cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6300b4d60 con 0x7fe6300d2360 2024-04-15T15:24:04.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3586894267 0 0) 0x7fe624001670 con 0x7fe6300d2360 2024-04-15T15:24:04.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe635adc640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 0x7fe6300b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:24:04.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe6362dd640 1 --2- 172.21.15.12:0/4187008293 >> v2:172.21.15.55:3300/0 conn(0x7fe6300d3270 0x7fe6300cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:24:04.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 >> v1:172.21.15.12:6789/0 conn(0x7fe6300d2360 legacy=0x7fe6300d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6180039b0 0x7fe618003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe618004800 0x7fe618004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 0x7fe6180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe6300b53b0 con 0x7fe6180039b0 2024-04-15T15:24:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe6300b4a80 con 0x7fe618004800 2024-04-15T15:24:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe61800bad0 con 0x7fe63015e700 2024-04-15T15:24:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe635adc640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6180039b0 0x7fe618003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe6352db640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe618004800 0x7fe618004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe6362dd640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 0x7fe6180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe635adc640 1 -- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 msgr2=0x7fe6180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:24:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe635adc640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe63015e700 0x7fe6180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe635adc640 1 -- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe618004800 msgr2=0x7fe618004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:24:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe635adc640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe618004800 0x7fe618004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe635adc640 1 -- 172.21.15.12:0/4187008293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe618003520 con 0x7fe6180039b0 2024-04-15T15:24:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.806+0000 7fe6352db640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe618004800 0x7fe618004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:24:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.807+0000 7fe635adc640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6180039b0 0x7fe618003dd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe6280098b0 tx=0x7fe618005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4578a150cf6ef2d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.807+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628034070 con 0x7fe6180039b0 2024-04-15T15:24:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.807+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe628030630 con 0x7fe6180039b0 2024-04-15T15:24:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.807+0000 7fe634ada640 1 -- 172.21.15.12:0/4187008293 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62800f040 con 0x7fe6180039b0 2024-04-15T15:24:04.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.808+0000 7fe637d40640 1 -- 172.21.15.12:0/4187008293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6180039b0 msgr2=0x7fe618003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.808+0000 7fe637d40640 1 --2- 172.21.15.12:0/4187008293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6180039b0 0x7fe618003dd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe6280098b0 tx=0x7fe618005830 comp rx=0 tx=0).stop 2024-04-15T15:24:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.808+0000 7fe6362dd640 1 -- 172.21.15.12:0/4187008293 reap_dead start 2024-04-15T15:24:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.808+0000 7fe637d40640 1 -- 172.21.15.12:0/4187008293 shutdown_connections 2024-04-15T15:24:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.808+0000 7fe637d40640 1 -- 172.21.15.12:0/4187008293 >> 172.21.15.12:0/4187008293 conn(0x7fe63008bf60 msgr2=0x7fe6300d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:24:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.808+0000 7fe637d40640 1 -- 172.21.15.12:0/4187008293 shutdown_connections 2024-04-15T15:24:04.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.808+0000 7fe637d40640 1 -- 172.21.15.12:0/4187008293 wait complete. 2024-04-15T15:24:04.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.809+0000 7fe637d40640 1 Processor -- start 2024-04-15T15:24:04.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.809+0000 7fe637d40640 1 -- start start 2024-04-15T15:24:04.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe637d40640 1 --2- >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6180039b0 0x7fe6300b9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe637d40640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6301f8de0 0x7fe63015cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe637d40640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6301f80f0 0x7fe6301f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe637d40640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe6300d1d40 con 0x7fe6301f80f0 2024-04-15T15:24:04.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe637d40640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe6300b4450 con 0x7fe6301f8de0 2024-04-15T15:24:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe637d40640 1 -- --> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] -- mon_getmap magic: 0 -- 0x7fe6300d1ec0 con 0x7fe6180039b0 2024-04-15T15:24:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6301f80f0 0x7fe6301f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6301f80f0 0x7fe6301f85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59126/0 (socket says 172.21.15.12:59126) 2024-04-15T15:24:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 -- 172.21.15.12:0/2096596524 learned_addr learned my addr 172.21.15.12:0/2096596524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-15T15:24:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe635adc640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6180039b0 0x7fe6300b9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6352db640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6301f8de0 0x7fe63015cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 -- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6180039b0 msgr2=0x7fe6300b9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6180039b0 0x7fe6300b9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 -- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6301f8de0 msgr2=0x7fe63015cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6301f8de0 0x7fe63015cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 -- 172.21.15.12:0/2096596524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6300b7220 con 0x7fe6301f80f0 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.810+0000 7fe6362dd640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6301f80f0 0x7fe6301f85a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe63008e870 tx=0x7fe620007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.811+0000 7fe62e7fc640 1 -- 172.21.15.12:0/2096596524 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620003ea0 con 0x7fe6301f80f0 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.811+0000 7fe62e7fc640 1 -- 172.21.15.12:0/2096596524 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe620019070 con 0x7fe6301f80f0 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.811+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe630160520 con 0x7fe6301f80f0 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.811+0000 7fe62e7fc640 1 -- 172.21.15.12:0/2096596524 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620014650 con 0x7fe6301f80f0 2024-04-15T15:24:04.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.811+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6301607a0 con 0x7fe6301f80f0 2024-04-15T15:24:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.812+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6300689e0 con 0x7fe6301f80f0 2024-04-15T15:24:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.812+0000 7fe62e7fc640 1 -- 172.21.15.12:0/2096596524 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe6200147f0 con 0x7fe6301f80f0 2024-04-15T15:24:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.813+0000 7fe62e7fc640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe61c06c980 0x7fe61c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:24:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.813+0000 7fe62e7fc640 1 -- 172.21.15.12:0/2096596524 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fe62008c030 con 0x7fe6301f80f0 2024-04-15T15:24:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.813+0000 7fe635adc640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe61c06c980 0x7fe61c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:24:04.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.813+0000 7fe635adc640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe61c06c980 0x7fe61c06ee40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe6280098b0 tx=0x7fe628031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:24:04.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:04.817+0000 7fe62e7fc640 1 -- 172.21.15.12:0/2096596524 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe630160520 con 0x7fe6301f80f0 2024-04-15T15:24:05.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.042+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6300ba500 con 0x7fe6301f80f0 2024-04-15T15:24:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.042+0000 7fe62e7fc640 1 -- 172.21.15.12:0/2096596524 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+3866 (secure 0 0 0) 0x7fe620059d30 con 0x7fe6301f80f0 2024-04-15T15:24:05.044 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-15T15:24:05.044 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/3031650058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6833","nonce":3031650058},{"type":"v1","addr":"172.21.15.12:6835","nonce":3031650058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6834/2772515949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":2772515949},{"type":"v1","addr":"172.21.15.12:6834","nonce":2772515949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5102,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/852515692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6835","nonce":852515692},{"type":"v1","addr":"172.21.15.55:6837","nonce":852515692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6836/2370870092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2370870092},{"type":"v1","addr":"172.21.15.55:6836","nonce":2370870092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[]} 2024-04-15T15:24:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 25 2024-04-15T15:24:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe61c06c980 msgr2=0x7fe61c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe61c06c980 0x7fe61c06ee40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe6280098b0 tx=0x7fe628031000 comp rx=0 tx=0).stop 2024-04-15T15:24:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6301f80f0 msgr2=0x7fe6301f85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:24:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6301f80f0 0x7fe6301f85a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe63008e870 tx=0x7fe620007700 comp rx=0 tx=0).stop 2024-04-15T15:24:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 shutdown_connections 2024-04-15T15:24:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6301f80f0 0x7fe6301f85a0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:6832/746784251,v1:172.21.15.55:6833/746784251] conn(0x7fe61c06c980 0x7fe61c06ee40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe6301f8de0 0x7fe63015cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 --2- 172.21.15.12:0/2096596524 >> [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] conn(0x7fe6180039b0 0x7fe6300b9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:24:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 >> 172.21.15.12:0/2096596524 conn(0x7fe63008bf60 msgr2=0x7fe6300d10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:24:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 shutdown_connections 2024-04-15T15:24:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-15T15:24:05.045+0000 7fe637d40640 1 -- 172.21.15.12:0/2096596524 wait complete. 2024-04-15T15:24:05.057 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-15T15:24:05.057 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-15T15:24:05.058 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-15T15:24:05.058 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.067 INFO:tasks.ceph.mds.a:Stopped 2024-04-15T15:24:05.067 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-15T15:24:05.067 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.116 INFO:tasks.ceph.mds.c:Stopped 2024-04-15T15:24:05.117 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-15T15:24:05.117 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.128 INFO:tasks.ceph.mds.b:Stopped 2024-04-15T15:24:05.129 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-15T15:24:05.129 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.181 INFO:tasks.ceph.mds.d:Stopped 2024-04-15T15:24:05.181 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-15T15:24:05.181 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-15T15:24:05.181 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.208 INFO:tasks.ceph.osd.0:Stopped 2024-04-15T15:24:05.208 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-15T15:24:05.208 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.283 INFO:tasks.ceph.osd.1:Stopped 2024-04-15T15:24:05.284 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-15T15:24:05.284 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.351 INFO:tasks.ceph.osd.2:Stopped 2024-04-15T15:24:05.352 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-15T15:24:05.352 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.419 INFO:tasks.ceph.osd.3:Stopped 2024-04-15T15:24:05.419 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-15T15:24:05.419 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.453 INFO:tasks.ceph.osd.4:Stopped 2024-04-15T15:24:05.453 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-15T15:24:05.453 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.525 INFO:tasks.ceph.osd.5:Stopped 2024-04-15T15:24:05.526 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-15T15:24:05.526 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.594 INFO:tasks.ceph.osd.6:Stopped 2024-04-15T15:24:05.594 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-15T15:24:05.594 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.663 INFO:tasks.ceph.osd.7:Stopped 2024-04-15T15:24:05.663 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-15T15:24:05.664 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-15T15:24:05.664 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.691 INFO:tasks.ceph.mgr.y:Stopped 2024-04-15T15:24:05.691 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-15T15:24:05.691 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.733 INFO:tasks.ceph.mgr.x:Stopped 2024-04-15T15:24:05.733 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-15T15:24:05.733 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-15T15:24:05.733 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.745 INFO:tasks.ceph.mon.a:Stopped 2024-04-15T15:24:05.745 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-15T15:24:05.746 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.785 INFO:tasks.ceph.mon.b:Stopped 2024-04-15T15:24:05.786 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-15T15:24:05.786 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:24:05.838 INFO:tasks.ceph.mon.c:Stopped 2024-04-15T15:24:05.838 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-15T15:24:05.838 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow request' | head -n 1 2024-04-15T15:24:05.899 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:24:05.900 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-15T15:24:06.096 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:24:06.096 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-15T15:24:06.179 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:24:06.179 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-15T15:24:06.237 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:24:06.237 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-15T15:24:06.296 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:24:06.296 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-15T15:24:06.450 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:24:06.450 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-15T15:24:06.533 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:24:06.533 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-15T15:24:06.616 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:24:06.616 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-15T15:24:06.672 INFO:tasks.ceph:Archiving mon data... 2024-04-15T15:24:06.673 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/data/mon.a.tgz 2024-04-15T15:24:06.673 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-04-15T15:24:06.690 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.XVMOOqm7Iu 2024-04-15T15:24:06.690 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.XVMOOqm7Iu 2024-04-15T15:24:06.832 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.XVMOOqm7Iu 2024-04-15T15:24:06.918 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.XVMOOqm7Iu is 360KB 2024-04-15T15:24:07.016 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.XVMOOqm7Iu 2024-04-15T15:24:07.031 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/data/mon.b.tgz 2024-04-15T15:24:07.031 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-04-15T15:24:07.050 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.nzUk5MvZei 2024-04-15T15:24:07.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.nzUk5MvZei 2024-04-15T15:24:07.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.nzUk5MvZei 2024-04-15T15:24:07.322 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.nzUk5MvZei is 482KB 2024-04-15T15:24:07.391 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.nzUk5MvZei 2024-04-15T15:24:07.405 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/data/mon.c.tgz 2024-04-15T15:24:07.405 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-04-15T15:24:07.460 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.VGRXdTlxFY 2024-04-15T15:24:07.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.VGRXdTlxFY 2024-04-15T15:24:07.617 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.VGRXdTlxFY 2024-04-15T15:24:07.703 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.VGRXdTlxFY is 482KB 2024-04-15T15:24:07.803 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.VGRXdTlxFY 2024-04-15T15:24:07.818 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-15T15:24:07.818 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-15T15:24:07.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-15T15:24:07.910 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-15T15:24:07.941 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-15T15:24:07.942 INFO:tasks.ceph:Archiving crash dumps... 2024-04-15T15:24:07.943 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/remote/smithi012/crash 2024-04-15T15:24:07.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-15T15:24:07.982 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/remote/smithi055/crash 2024-04-15T15:24:07.983 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-15T15:24:08.011 INFO:tasks.ceph:Shutting down logrotate 2024-04-15T15:24:08.011 DEBUG:teuthology.orchestra.run.smithi012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:24:08.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:24:08.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-15T15:24:08.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-15T15:24:08.157 INFO:tasks.ceph:Compressing logs... 2024-04-15T15:24:08.157 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T15:24:08.173 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T15:24:08.199 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52229.log 2024-04-15T15:24:08.249 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-15T15:24:08.249 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44424.log 2024-04-15T15:24:08.249 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37632.log 2024-04-15T15:24:08.249 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.1.44424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37592.log 2024-04-15T15:24:08.249 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37632.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-15T15:24:08.250 INFO:teuthology.orchestra.run.smithi055.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37632.log.gz 2024-04-15T15:24:08.250 INFO:teuthology.orchestra.run.smithi055.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.1.44424.log.gz 2024-04-15T15:24:08.250 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37432.log 2024-04-15T15:24:08.250 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37592.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-15T15:24:08.250 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37592.log.gz 2024-04-15T15:24:08.250 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.46473.log 2024-04-15T15:24:08.250 INFO:teuthology.orchestra.run.smithi055.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37432.log.gz 2024-04-15T15:24:08.250 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45624.log 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.3.46473.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37472.log 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.1.45624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37712.log 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37472.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37472.log.gz 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.1.45624.log.gz 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.3.46473.log.gz 2024-04-15T15:24:08.251 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37712.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37712.log.gz 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37552.log 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37512.log 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44881.log 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1%/var/log/ceph/ceph-client.admin.37512.log: -- replaced with /var/log/ceph/ceph-client.admin.37552.log.gz 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37512.log.gz 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr: /var/log/ceph/ceph-mgr.x.log 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.3.44881.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-15T15:24:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose 91.2% -- /var/log/ceph/ceph-client.1.46132.log 2024-04-15T15:24:08.253 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.3.44881.log.gz 2024-04-15T15:24:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph.log: 92.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-15T15:24:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-15T15:24:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.1.46132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37672.log 2024-04-15T15:24:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mon.b.log: 91.2% -- replaced with /var/log/ceph/ceph-client.1.46132.log.gz 2024-04-15T15:24:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-15T15:24:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37672.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37672.log.gz 2024-04-15T15:24:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-15T15:24:08.262 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 92.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-15T15:24:08.302 INFO:teuthology.orchestra.run.smithi055.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-15T15:24:08.342 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56174.log 2024-04-15T15:24:08.343 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46519.log 2024-04-15T15:24:08.343 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53337.log 2024-04-15T15:24:08.343 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52229.log.gz 2024-04-15T15:24:08.343 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56174.log.gz 2024-04-15T15:24:08.343 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50591.log 2024-04-15T15:24:08.343 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53337.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46519.log.gz 2024-04-15T15:24:08.343 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55750.log 2024-04-15T15:24:08.344 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53337.log.gz 2024-04-15T15:24:08.344 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49448.log 2024-04-15T15:24:08.344 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50591.log.gz 2024-04-15T15:24:08.344 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45427.log 2024-04-15T15:24:08.344 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49448.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55750.log.gz 2024-04-15T15:24:08.344 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-15T15:24:08.344 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49448.log.gz 2024-04-15T15:24:08.344 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59366.log 2024-04-15T15:24:08.345 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45427.log.gz 2024-04-15T15:24:08.345 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52895.log 2024-04-15T15:24:08.345 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54896.log 2024-04-15T15:24:08.345 INFO:teuthology.orchestra.run.smithi012.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59366.log.gz 2024-04-15T15:24:08.345 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51767.log 2024-04-15T15:24:08.345 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52895.log.gz 2024-04-15T15:24:08.345 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49925.log 2024-04-15T15:24:08.345 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51725.log 2024-04-15T15:24:08.346 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54896.log.gz 2024-04-15T15:24:08.346 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51767.log.gz 2024-04-15T15:24:08.346 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54980.log 2024-04-15T15:24:08.346 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49925.log.gz 2024-04-15T15:24:08.346 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53295.log 2024-04-15T15:24:08.346 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54980.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51725.log.gz 2024-04-15T15:24:08.346 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47065.log 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54980.log.gz 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46181.log 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53295.log.gz 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50339.log 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46181.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47065.log.gz 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48986.log 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46181.log.gz 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51389.log 2024-04-15T15:24:08.347 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50339.log.gz 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57460.log 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51389.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48986.log.gz 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46897.log 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51389.log.gz 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56868.log 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57460.log.gz 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43284.log 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56868.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.46897.log.gz -5 2024-04-15T15:24:08.348 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51893.log 2024-04-15T15:24:08.349 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56868.log.gz 2024-04-15T15:24:08.349 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57586.log 2024-04-15T15:24:08.349 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43284.log.gz 2024-04-15T15:24:08.349 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50093.log 2024-04-15T15:24:08.349 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57586.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51893.log.gz 2024-04-15T15:24:08.349 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51515.log 2024-04-15T15:24:08.349 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57586.log.gz 2024-04-15T15:24:08.349 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52355.log 2024-04-15T15:24:08.350 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50093.log.gz 2024-04-15T15:24:08.350 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48482.log 2024-04-15T15:24:08.350 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52355.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51515.log.gz -5 2024-04-15T15:24:08.350 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46393.log 2024-04-15T15:24:08.350 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52355.log.gz 2024-04-15T15:24:08.350 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48776.log 2024-04-15T15:24:08.350 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48482.log.gz 2024-04-15T15:24:08.351 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48246.log 2024-04-15T15:24:08.351 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48776.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46393.log.gz 2024-04-15T15:24:08.351 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54466.log 2024-04-15T15:24:08.351 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48776.log.gz 2024-04-15T15:24:08.351 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51935.log 2024-04-15T15:24:08.351 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48246.log.gz 2024-04-15T15:24:08.351 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56988.log 2024-04-15T15:24:08.351 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51935.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54466.log.gz 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43065.log 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51935.log.gz 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51599.log 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56988.log.gz 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46197.log 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51599.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43065.log.gz 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50833.log 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51599.log.gz 2024-04-15T15:24:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53127.log 2024-04-15T15:24:08.353 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46197.log.gz 2024-04-15T15:24:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46435.log 2024-04-15T15:24:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53127.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50833.log.gz 2024-04-15T15:24:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59156.log 2024-04-15T15:24:08.353 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53127.log.gz 2024-04-15T15:24:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50381.log 2024-04-15T15:24:08.353 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46435.log.gz 2024-04-15T15:24:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50213.log 2024-04-15T15:24:08.354 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50381.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59156.log.gz 2024-04-15T15:24:08.354 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52811.log 2024-04-15T15:24:08.354 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50381.log.gz 2024-04-15T15:24:08.354 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44941.log 2024-04-15T15:24:08.354 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50213.log.gz 2024-04-15T15:24:08.354 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57114.log 2024-04-15T15:24:08.354 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44941.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52811.log.gz 2024-04-15T15:24:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50297.log 2024-04-15T15:24:08.355 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44941.log.gz 2024-04-15T15:24:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52271.log 2024-04-15T15:24:08.355 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57114.log.gz 2024-04-15T15:24:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55106.log 2024-04-15T15:24:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50297.log: /var/log/ceph/ceph-client.admin.52271.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50297.log.gz 2024-04-15T15:24:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50749.log 2024-04-15T15:24:08.355 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52271.log.gz 2024-04-15T15:24:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58132.log 2024-04-15T15:24:08.356 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55106.log.gz 2024-04-15T15:24:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45856.log 2024-04-15T15:24:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58132.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50749.log.gz 2024-04-15T15:24:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43711.log 2024-04-15T15:24:08.356 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58132.log.gz 2024-04-15T15:24:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59550.log 2024-04-15T15:24:08.356 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45856.log.gz 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54776.log 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59550.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43711.log.gz 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46603.log 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59550.log.gz 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56742.log 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54776.log.gz 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58903.log 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49783.log 2024-04-15T15:24:08.357 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46603.log.gz 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56742.log.gz 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44071.log 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58903.log.gz 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54938.log 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44071.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49783.log.gz 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46939.log 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44071.log.gz 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49112.log 2024-04-15T15:24:08.358 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54938.log.gz 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48524.log 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49112.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46939.log.gz 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45335.log 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49112.log.gz 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57256.log 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48524.log.gz 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51101.log 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57256.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45335.log.gz 2024-04-15T15:24:08.359 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52769.log 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57256.log.gz 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46687.log 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51101.log.gz 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50791.log 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46687.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52769.log.gz 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48902.log 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46687.log.gz 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52397.log 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48902.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50791.log.gz 2024-04-15T15:24:08.360 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46561.log 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52397.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48902.log.gz 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43963.log 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52397.log.gz/var/log/ceph/ceph-client.admin.46561.log: 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58988.log 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46561.log.gz 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43667.log 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58988.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43963.log.gz 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58817.log 2024-04-15T15:24:08.361 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58988.log.gz 2024-04-15T15:24:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45861.log 2024-04-15T15:24:08.362 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43667.log.gz 2024-04-15T15:24:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57298.log 2024-04-15T15:24:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45861.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58817.log.gz 2024-04-15T15:24:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53085.log 2024-04-15T15:24:08.362 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45861.log.gz 2024-04-15T15:24:08.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52685.log 2024-04-15T15:24:08.362 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57298.log.gz 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54198.log 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49406.log 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53085.log.gz 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52685.log.gz 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49028.log 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54198.log.gz 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51263.log 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49070.log 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49406.log.gz 2024-04-15T15:24:08.363 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49028.log.gz 2024-04-15T15:24:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57922.log 2024-04-15T15:24:08.364 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51263.log.gz 2024-04-15T15:24:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48650.log 2024-04-15T15:24:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57922.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49070.log.gz 2024-04-15T15:24:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53781.log 2024-04-15T15:24:08.364 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57922.log.gz 2024-04-15T15:24:08.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54692.log 2024-04-15T15:24:08.364 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4%/var/log/ceph/ceph-client.admin.53781.log: -- replaced with /var/log/ceph/ceph-client.admin.48650.log.gz 2024-04-15T15:24:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49654.log 2024-04-15T15:24:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54692.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53781.log.gz 2024-04-15T15:24:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45109.log 2024-04-15T15:24:08.365 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54692.log.gz 2024-04-15T15:24:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54240.log 2024-04-15T15:24:08.365 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49654.log.gz 2024-04-15T15:24:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43921.log 2024-04-15T15:24:08.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54240.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45109.log.gz 2024-04-15T15:24:08.366 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58725.log 2024-04-15T15:24:08.366 INFO:teuthology.orchestra.run.smithi012.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54240.log.gz 2024-04-15T15:24:08.366 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58234.log 2024-04-15T15:24:08.366 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43921.log.gz 2024-04-15T15:24:08.366 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52558.log 2024-04-15T15:24:08.366 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58725.log: /var/log/ceph/ceph-client.admin.58234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49322.log 2024-04-15T15:24:08.366 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58725.log.gz 2024-04-15T15:24:08.366 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58234.log.gz 2024-04-15T15:24:08.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50009.log 2024-04-15T15:24:08.367 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52558.log.gz 2024-04-15T15:24:08.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59240.log 2024-04-15T15:24:08.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50009.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49322.log.gz 2024-04-15T15:24:08.367 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58402.log 2024-04-15T15:24:08.367 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50009.log.gz 2024-04-15T15:24:08.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44983.log 2024-04-15T15:24:08.367 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59240.log.gz 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50549.log 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44983.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58402.log.gz 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54818.log 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44983.log.gz 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46771.log 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50549.log.gz 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49883.log 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46771.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54818.log.gz 2024-04-15T15:24:08.368 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45679.log 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46771.log.gz 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56425.log 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49883.log.gz 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58556.log 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56425.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45679.log.gz 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51017.log 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56425.log.gz 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44883.log 2024-04-15T15:24:08.369 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58556.log.gz 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51017.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44883.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51017.log.gz 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58006.log 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44883.log.gz 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48288.log 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47235.log 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48288.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58006.log.gz 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56359.log 2024-04-15T15:24:08.370 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48288.log.gz 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52313.log 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47235.log: /var/log/ceph/ceph-client.admin.56359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50051.log 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47235.log.gz 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52313.log: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46189.log 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.56359.log.gz 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52313.log.gz 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50171.log 2024-04-15T15:24:08.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50051.log: /var/log/ceph/ceph-client.admin.46189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58318.log 2024-04-15T15:24:08.372 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50051.log.gz 2024-04-15T15:24:08.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50171.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46189.log.gz 2024-04-15T15:24:08.372 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49967.log 2024-04-15T15:24:08.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58318.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50171.log.gz 2024-04-15T15:24:08.372 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45855.log 2024-04-15T15:24:08.372 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58318.log.gz 2024-04-15T15:24:08.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45151.log 2024-04-15T15:24:08.372 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49967.log.gz 2024-04-15T15:24:08.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45857.log 2024-04-15T15:24:08.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45151.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45855.log.gz 2024-04-15T15:24:08.373 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46186.log 2024-04-15T15:24:08.373 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2024-04-15T15:24:08.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55367.log 2024-04-15T15:24:08.373 INFO:teuthology.orchestra.run.smithi012.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45857.log.gz 2024-04-15T15:24:08.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49364.log 2024-04-15T15:24:08.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55367.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46186.log.gz 2024-04-15T15:24:08.374 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58945.log 2024-04-15T15:24:08.374 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55367.log.gz 2024-04-15T15:24:08.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58598.log 2024-04-15T15:24:08.374 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49364.log.gz 2024-04-15T15:24:08.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55232.log 2024-04-15T15:24:08.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58598.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58945.log.gz 2024-04-15T15:24:08.374 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53463.log 2024-04-15T15:24:08.374 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58598.log.gz 2024-04-15T15:24:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50465.log 2024-04-15T15:24:08.375 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55232.log.gz 2024-04-15T15:24:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57156.log 2024-04-15T15:24:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43161.log 2024-04-15T15:24:08.375 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53463.log.gz 2024-04-15T15:24:08.375 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50465.log.gz 2024-04-15T15:24:08.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59508.log 2024-04-15T15:24:08.375 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57156.log.gz 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59282.log 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59508.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43161.log.gz 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45860.log 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59508.log.gz 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37405.log 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59282.log.gz 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56553.log 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37405.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45860.log.gz 2024-04-15T15:24:08.376 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51305.log 2024-04-15T15:24:08.377 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37405.log.gz 2024-04-15T15:24:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46729.log 2024-04-15T15:24:08.377 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56553.log.gz 2024-04-15T15:24:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37508.log 2024-04-15T15:24:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46729.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51305.log.gz 2024-04-15T15:24:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45511.log 2024-04-15T15:24:08.377 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46729.log.gz 2024-04-15T15:24:08.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46194.log 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37508.log.gz 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50507.log 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46194.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45511.log.gz 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56826.log 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46194.log.gz 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59468.log 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50507.log.gz 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45553.log 2024-04-15T15:24:08.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59468.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56826.log.gz 2024-04-15T15:24:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57880.log 2024-04-15T15:24:08.379 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59468.log.gz 2024-04-15T15:24:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56467.log 2024-04-15T15:24:08.379 INFO:teuthology.orchestra.run.smithi012.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45553.log.gz 2024-04-15T15:24:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56910.log 2024-04-15T15:24:08.379 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57382.log 2024-04-15T15:24:08.379 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57880.log.gz 2024-04-15T15:24:08.379 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56467.log.gz 2024-04-15T15:24:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46190.log 2024-04-15T15:24:08.380 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56910.log.gz 2024-04-15T15:24:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54366.log 2024-04-15T15:24:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46190.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57382.log.gz 2024-04-15T15:24:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49238.log 2024-04-15T15:24:08.380 INFO:teuthology.orchestra.run.smithi012.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46190.log.gz 2024-04-15T15:24:08.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54366.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-15T15:24:08.380 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54366.log.gz 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58090.log 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.1.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49238.log.gz 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54508.log 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45859.log 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54508.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58090.log.gz 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51143.log 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54508.log.gz 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48734.log 2024-04-15T15:24:08.381 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45859.log.gz 2024-04-15T15:24:08.382 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45247.log 2024-04-15T15:24:08.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51143.log: /var/log/ceph/ceph-client.admin.48734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49586.log 2024-04-15T15:24:08.382 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51143.log.gz 2024-04-15T15:24:08.382 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48734.log.gz 2024-04-15T15:24:08.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48860.log 2024-04-15T15:24:08.382 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53169.log 2024-04-15T15:24:08.382 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45247.log.gz 2024-04-15T15:24:08.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49586.log: /var/log/ceph/ceph-client.admin.48860.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49586.log.gz 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43411.log 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48860.log.gz 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57628.log 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53169.log: /var/log/ceph/ceph-client.admin.43411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52061.log 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53169.log.gz 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57628.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43411.log.gz 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.56052.log 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57628.log.gz 2024-04-15T15:24:08.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51347.log 2024-04-15T15:24:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48440.log 2024-04-15T15:24:08.384 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52061.log.gz 2024-04-15T15:24:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.2.56052.log: /var/log/ceph/ceph-client.admin.51347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52477.log 2024-04-15T15:24:08.384 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51347.log.gz 2024-04-15T15:24:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58276.log 2024-04-15T15:24:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56597.log 2024-04-15T15:24:08.384 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48440.log.gz 2024-04-15T15:24:08.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52477.log: /var/log/ceph/ceph-client.admin.58276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52853.log 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52477.log.gz 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.2.56052.log.gz 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58276.log.gz 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58360.log 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52600.log 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7%/var/log/ceph/ceph-client.admin.52853.log: -- replaced with /var/log/ceph/ceph-client.admin.56597.log.gz 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52187.log 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52853.log.gz 2024-04-15T15:24:08.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43242.log 2024-04-15T15:24:08.386 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58360.log.gz 2024-04-15T15:24:08.386 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52600.log.gz 2024-04-15T15:24:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52187.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31124.log 2024-04-15T15:24:08.386 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52187.log.gz 2024-04-15T15:24:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42966.log 2024-04-15T15:24:08.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43242.log: /var/log/ceph/ceph.tmp-client.admin.31124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46193.log 2024-04-15T15:24:08.386 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31124.log.gz 2024-04-15T15:24:08.386 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43242.log.gz 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52642.log 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42966.log.gz 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54550.log 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46193.log.gz 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47626.log 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52642.log.gz 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53589.log 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54550.log: /var/log/ceph/ceph-client.0.47626.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54550.log.gz 2024-04-15T15:24:08.387 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57838.log 2024-04-15T15:24:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2024-04-15T15:24:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53589.log: 90.4% -- replaced with /var/log/ceph/ceph-client.0.47626.log.gz 2024-04-15T15:24:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54041.log 2024-04-15T15:24:08.388 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53589.log.gz 2024-04-15T15:24:08.388 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57838.log.gz 2024-04-15T15:24:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48692.log 2024-04-15T15:24:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43583.log: /var/log/ceph/ceph-client.0.54041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46981.log 2024-04-15T15:24:08.388 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52937.log 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48692.log.gz 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46981.log: 89.0% -- replaced with /var/log/ceph/ceph-client.0.54041.log.gz 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53505.log 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46855.log 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46981.log.gz 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52937.log: /var/log/ceph/ceph-client.admin.53505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56511.log 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52937.log.gz 2024-04-15T15:24:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46855.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53505.log.gz 2024-04-15T15:24:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-15T15:24:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47107.log 2024-04-15T15:24:08.390 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46855.log.gz 2024-04-15T15:24:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56511.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53253.log 2024-04-15T15:24:08.390 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56511.log.gz 2024-04-15T15:24:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45067.log 2024-04-15T15:24:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47107.log: /var/log/ceph/ceph-client.admin.53253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53631.log 2024-04-15T15:24:08.390 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53253.log.gz 2024-04-15T15:24:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45067.log: gzip 91.3% -- replaced with /var/log/ceph/ceph-client.admin.47107.log.gz 2024-04-15T15:24:08.391 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53673.log 2024-04-15T15:24:08.391 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45067.log.gz 2024-04-15T15:24:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57670.log 2024-04-15T15:24:08.391 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53631.log.gz 2024-04-15T15:24:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43327.log 2024-04-15T15:24:08.391 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53673.log.gz 2024-04-15T15:24:08.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57670.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57670.log.gz 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr: -- /var/log/ceph/ceph-client.admin.51809.log 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45850.log 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51809.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43327.log.gz 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52019.log 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51809.log.gz 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52145.log 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45850.log.gz 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59072.log 2024-04-15T15:24:08.392 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52019.log.gz 2024-04-15T15:24:08.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50423.log 2024-04-15T15:24:08.393 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52145.log.gz 2024-04-15T15:24:08.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53547.log 2024-04-15T15:24:08.393 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53421.log 2024-04-15T15:24:08.393 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4%/var/log/ceph/ceph-client.admin.50423.log: -- replaced with /var/log/ceph/ceph-client.admin.59072.log.gz 2024-04-15T15:24:08.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37152.log 2024-04-15T15:24:08.393 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50423.log.gz 2024-04-15T15:24:08.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53421.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53547.log.gz 2024-04-15T15:24:08.394 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43795.log 2024-04-15T15:24:08.394 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53421.log.gz 2024-04-15T15:24:08.394 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43879.log 2024-04-15T15:24:08.394 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49825.log 2024-04-15T15:24:08.394 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43795.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.37152.log.gz/var/log/ceph/ceph-client.admin.43879.log: 2024-04-15T15:24:08.394 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43495.log 2024-04-15T15:24:08.394 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43795.log.gz 2024-04-15T15:24:08.394 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43879.log.gz 2024-04-15T15:24:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46645.log 2024-04-15T15:24:08.395 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49825.log.gz 2024-04-15T15:24:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51185.log 2024-04-15T15:24:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.admin.51557.log 2024-04-15T15:24:08.395 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43495.log.gz 2024-04-15T15:24:08.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46645.log: /var/log/ceph/ceph-client.admin.51185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51431.log 2024-04-15T15:24:08.395 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46645.log.gz 2024-04-15T15:24:08.395 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2%/var/log/ceph/ceph-client.admin.51557.log: -- replaced with /var/log/ceph/ceph-client.admin.51185.log.gz 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59114.log 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51557.log.gz 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57214.log 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.51431.log.gz -5 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44715.log 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59114.log: /var/log/ceph/ceph-client.admin.57214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45637.log 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59114.log.gz 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44715.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57214.log.gz 2024-04-15T15:24:08.396 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50917.log 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44715.log.gz 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57072.log 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43625.log 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45637.log.gz 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50917.log: /var/log/ceph/ceph-client.admin.57072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44841.log 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50917.log.gz 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43625.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57072.log.gz 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48818.log 2024-04-15T15:24:08.397 INFO:teuthology.orchestra.run.smithi012.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43625.log.gz 2024-04-15T15:24:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45852.log 2024-04-15T15:24:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53211.log 2024-04-15T15:24:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48818.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44841.log.gz 2024-04-15T15:24:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52103.log 2024-04-15T15:24:08.398 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48818.log.gz 2024-04-15T15:24:08.398 INFO:teuthology.orchestra.run.smithi012.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45852.log.gz 2024-04-15T15:24:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43369.log 2024-04-15T15:24:08.398 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53211.log.gz 2024-04-15T15:24:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45595.log 2024-04-15T15:24:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55064.log 2024-04-15T15:24:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43369.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52103.log.gz 2024-04-15T15:24:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58048.log 2024-04-15T15:24:08.399 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43369.log.gz 2024-04-15T15:24:08.399 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45595.log.gz 2024-04-15T15:24:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58192.log 2024-04-15T15:24:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55064.log.gz 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58048.log: /var/log/ceph/ceph-client.admin.58192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48398.log 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58048.log.gz 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58192.log.gz 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49280.log 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50875.log 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48398.log: /var/log/ceph/ceph-client.admin.49280.log: gzip 88.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.57964.log 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48398.log.gz 2024-04-15T15:24:08.400 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49280.log.gz 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46477.log 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3%/var/log/ceph/ceph-client.admin.57964.log: -- replaced with /var/log/ceph/ceph-client.admin.50875.log.gz 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58767.log 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57964.log.gz 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46477.log: gzip -5 --verbose -- 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46477.log.gz 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.47748.log 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55792.log 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58767.log.gz 2024-04-15T15:24:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54734.log 2024-04-15T15:24:08.402 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47748.log.gz 2024-04-15T15:24:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47023.log 2024-04-15T15:24:08.402 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55792.log.gz 2024-04-15T15:24:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45469.log 2024-04-15T15:24:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47023.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54734.log.gz 2024-04-15T15:24:08.402 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52727.log 2024-04-15T15:24:08.402 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47023.log.gz 2024-04-15T15:24:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45469.log: gzip -5 --verbose 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45469.log.gz 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr: -- /var/log/ceph/ceph-client.admin.54424.log 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52727.log: 93.0%gzip -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51683.log 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52727.log.gz 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44481.log 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54424.log.gz 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46196.log 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51683.log.gz 2024-04-15T15:24:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43837.log 2024-04-15T15:24:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54324.log 2024-04-15T15:24:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43837.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46196.log.gz 2024-04-15T15:24:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58640.log 2024-04-15T15:24:08.404 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43837.log.gz 2024-04-15T15:24:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49154.log 2024-04-15T15:24:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58640.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55190.log 2024-04-15T15:24:08.404 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54324.log.gz 2024-04-15T15:24:08.404 INFO:teuthology.orchestra.run.smithi012.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.44481.log.gz 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58640.log.gz 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57502.log 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49154.log.gz 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56317.log 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57030.log 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57502.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55190.log.gz 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56317.log: gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57502.log.gz 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50707.log 2024-04-15T15:24:08.405 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56317.log.gz 2024-04-15T15:24:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54650.log 2024-04-15T15:24:08.406 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57030.log.gz 2024-04-15T15:24:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51977.log 2024-04-15T15:24:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50707.log: /var/log/ceph/ceph-client.admin.54650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57712.log 2024-04-15T15:24:08.406 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50707.log.gz 2024-04-15T15:24:08.406 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54650.log.gz 2024-04-15T15:24:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51977.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-15T15:24:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43539.log 2024-04-15T15:24:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57712.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51977.log.gz 2024-04-15T15:24:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59408.log 2024-04-15T15:24:08.407 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57712.log.gz 2024-04-15T15:24:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51473.log 2024-04-15T15:24:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59408.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43539.log.gz 2024-04-15T15:24:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52979.log 2024-04-15T15:24:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55148.log 2024-04-15T15:24:08.407 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59408.log.gz 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58682.log 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51473.log.gz 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52979.log.gz 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53715.log 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55148.log: /var/log/ceph/ceph-client.admin.58682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48944.log 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1%/var/log/ceph/ceph-client.admin.53715.log: gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.58682.log.gz -- 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.49490.log 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55148.log.gz 2024-04-15T15:24:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57796.log 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48944.log: /var/log/ceph/ceph-client.admin.49490.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53715.log.gz 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4%/var/log/ceph/ceph-client.admin.57796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57340.log 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48944.log.gz 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49490.log.gz 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57796.log.gz 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49196.log 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55274.log 2024-04-15T15:24:08.409 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-15T15:24:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49196.log: -- replaced with /var/log/ceph/ceph-client.admin.57340.log.gz 2024-04-15T15:24:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55274.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56639.log 2024-04-15T15:24:08.410 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49196.log.gz 2024-04-15T15:24:08.410 INFO:teuthology.orchestra.run.smithi012.stderr: 85.5% -- replaced with /var/log/ceph/ceph.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55274.log.gz 2024-04-15T15:24:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.2.log: 2024-04-15T15:24:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-15T15:24:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57544.log 2024-04-15T15:24:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55627.log 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.56639.log.gz 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57544.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57544.log.gz 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.55627.log: 91.8% -- replaced with /var/log/ceph/ceph-client.0.55627.log.gz 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56784.log 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49698.log 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44757.log: gzip/var/log/ceph/ceph-client.admin.56784.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45025.log 2024-04-15T15:24:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49698.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56784.log.gz 2024-04-15T15:24:08.412 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2024-04-15T15:24:08.412 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49698.log.gz 2024-04-15T15:24:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48608.log 2024-04-15T15:24:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45025.log: gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45025.log.gz 2024-04-15T15:24:08.412 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.45290.log 2024-04-15T15:24:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51641.log 2024-04-15T15:24:08.412 INFO:teuthology.orchestra.run.smithi012.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48608.log.gz 2024-04-15T15:24:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59198.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45290.log.gz 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44005.log 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51641.log.gz 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59198.log: gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59198.log.gz 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.57754.log 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56251.log 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43753.log 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44005.log.gz 2024-04-15T15:24:08.413 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57754.log.gz 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48566.log 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45384.log 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48566.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43753.log.gz 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56251.log.gz 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48566.log.gz 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55022.log 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53379.log 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45384.log: /var/log/ceph/ceph-client.admin.55022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53023.log 2024-04-15T15:24:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53379.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45384.log.gz 2024-04-15T15:24:08.415 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55022.log.gz 2024-04-15T15:24:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51059.log 2024-04-15T15:24:08.415 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53379.log.gz 2024-04-15T15:24:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46813.log 2024-04-15T15:24:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53023.log: /var/log/ceph/ceph-client.admin.51059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50255.log 2024-04-15T15:24:08.415 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53023.log.gz 2024-04-15T15:24:08.415 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51059.log.gz 2024-04-15T15:24:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46813.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46813.log.gz 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50633.log 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59324.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50255.log.gz 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50975.log 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50633.log.gz 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.59324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48124.log 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59324.log.gz 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44673.log 2024-04-15T15:24:08.416 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50975.log.gz 2024-04-15T15:24:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58859.log 2024-04-15T15:24:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.2.48124.log: /var/log/ceph/ceph-client.admin.44673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51851.log 2024-04-15T15:24:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58859.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44673.log.gz 2024-04-15T15:24:08.417 INFO:teuthology.orchestra.run.smithi012.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58859.log.gz 2024-04-15T15:24:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44799.log 2024-04-15T15:24:08.417 INFO:teuthology.orchestra.run.smithi012.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.2.48124.log.gz 2024-04-15T15:24:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51851.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51851.log.gz 2024-04-15T15:24:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59030.log 2024-04-15T15:24:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44799.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-15T15:24:08.418 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6%/var/log/ceph/ceph-client.admin.59030.log: -- replaced with /var/log/ceph/ceph-client.admin.44799.log.gz 2024-04-15T15:24:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43453.log 2024-04-15T15:24:08.418 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59030.log.gz 2024-04-15T15:24:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54282.log 2024-04-15T15:24:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.43453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54592.log 2024-04-15T15:24:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54282.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43453.log.gz 2024-04-15T15:24:08.418 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54282.log.gz 2024-04-15T15:24:08.419 INFO:teuthology.orchestra.run.smithi012.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-15T15:24:08.419 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49740.log 2024-04-15T15:24:08.419 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54592.log: /var/log/ceph/ceph-client.admin.49740.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54592.log.gz 2024-04-15T15:24:08.419 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49740.log.gz 2024-04-15T15:24:08.420 INFO:teuthology.orchestra.run.smithi055.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-15T15:24:08.420 INFO:teuthology.orchestra.run.smithi055.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-15T15:24:08.420 INFO:teuthology.orchestra.run.smithi055.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-15T15:24:08.452 INFO:teuthology.orchestra.run.smithi012.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-15T15:24:08.933 INFO:teuthology.orchestra.run.smithi055.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-15T15:24:08.942 INFO:teuthology.orchestra.run.smithi055.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-15T15:24:08.943 INFO:teuthology.orchestra.run.smithi012.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-15T15:24:08.947 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-15T15:24:08.954 INFO:teuthology.orchestra.run.smithi012.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-15T15:24:08.958 INFO:teuthology.orchestra.run.smithi055.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-15T15:24:08.967 INFO:teuthology.orchestra.run.smithi012.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-15T15:24:08.969 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-15T15:24:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m0.784s 2024-04-15T15:24:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m3.048s 2024-04-15T15:24:08.969 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m0.291s 2024-04-15T15:24:08.974 INFO:teuthology.orchestra.run.smithi055.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-15T15:24:08.976 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-04-15T15:24:08.976 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m0.764s 2024-04-15T15:24:08.976 INFO:teuthology.orchestra.run.smithi055.stderr:user 0m3.283s 2024-04-15T15:24:08.976 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m0.146s 2024-04-15T15:24:08.977 INFO:tasks.ceph:Archiving logs... 2024-04-15T15:24:08.977 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/remote/smithi012/log 2024-04-15T15:24:08.978 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T15:24:09.723 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/remote/smithi055/log 2024-04-15T15:24:09.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T15:24:10.180 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-15T15:24:10.191 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-15T15:24:10.192 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T15:24:10.194 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T15:24:10.290 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T15:24:10.299 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T15:24:10.299 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:24:10.302 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:24:10.316 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:24:10.318 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:24:10.321 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:24:10.321 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-04-15T15:24:10.321 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv01.front.sepia.ceph.com 3 6 377 58 +1644us[+1805us] +/- 40ms 2024-04-15T15:24:10.321 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 59 -1684us[-1524us] +/- 55ms 2024-04-15T15:24:10.321 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -643us[ -482us] +/- 79ms 2024-04-15T15:24:10.321 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:24:10.323 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:24:10.324 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-04-15T15:24:10.324 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv01.front.sepia.ceph.com 3 6 377 59 +1645us[+1816us] +/- 40ms 2024-04-15T15:24:10.324 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 58 -1676us[-1676us] +/- 55ms 2024-04-15T15:24:10.324 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -625us[ -625us] +/- 79ms 2024-04-15T15:24:10.324 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:24:10.325 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T15:24:10.337 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T15:24:10.338 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T15:24:10.367 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T15:24:10.450 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T15:24:10.528 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:24:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:24:10.562 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:24:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:24:10.595 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T15:24:10.604 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T15:24:10.627 INFO:teuthology.task.internal:Duration was 653.102935 seconds 2024-04-15T15:24:10.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T15:24:10.657 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T15:24:10.657 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T15:24:10.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T15:24:10.704 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:24:10.708 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:24:11.010 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-15T15:24:11.010 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-04-15T15:24:11.011 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T15:24:11.031 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2024-04-15T15:24:11.032 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T15:24:11.055 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-15T15:24:11.056 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T15:24:11.075 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T15:24:11.126 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-15T15:24:11.126 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T15:24:11.145 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T15:24:11.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T15:24:11.281 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T15:24:11.282 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T15:24:11.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T15:24:11.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T15:24:11.348 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:24:11.353 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:24:11.377 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-04-15T15:24:11.404 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2024-04-15T15:24:11.421 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:24:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:24:11.449 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:24:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:24:11.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T15:24:11.487 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T15:24:11.487 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/remote/smithi012 2024-04-15T15:24:11.488 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T15:24:11.529 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144/remote/smithi055 2024-04-15T15:24:11.529 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T15:24:11.566 INFO:teuthology.task.internal:Removing archive directory... 2024-04-15T15:24:11.567 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T15:24:11.569 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T15:24:11.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T15:24:11.628 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T15:24:11.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T15:24:11.652 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T15:24:11.653 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T15:24:11.655 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T15:24:11.667 INFO:teuthology.orchestra.run.smithi012.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 15:24 /home/ubuntu/cephtest 2024-04-15T15:24:11.671 INFO:teuthology.orchestra.run.smithi055.stdout: 265317 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 15:24 /home/ubuntu/cephtest 2024-04-15T15:24:11.672 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T15:24:11.709 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T15:24:11.775 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T15:24:11.802 DEBUG:teuthology.nuke:shortname: smithi012 2024-04-15T15:24:11.803 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:24:11.827 DEBUG:teuthology.nuke:shortname: smithi055 2024-04-15T15:24:11.827 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:24:11.831 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 15:04:59.841715', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIE9jX1BIR5LrfW9u7p9ZFWARk8rZM4rxQ7Pj9xRDsPwJAkFNbl74+1mqPiy2sFSYQdawFpR9IBX0HvGSLEKDHk='} 2024-04-15T15:24:11.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:54:08-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657144', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 15:04:59.843113', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxUayo0doJBrH/sTt93Zv9BrOxzjdCJzWs0DlBISUHJ16DVvUvYhyj5tpb4iCkQc063omFSnO4e+x1yQTn2nKg='} 2024-04-15T15:24:11.883 INFO:teuthology.orchestra.console.smithi012:Power off 2024-04-15T15:24:11.884 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T15:24:11.913 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2024-04-15T15:24:11.913 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:24:11.931 INFO:teuthology.orchestra.console.smithi055:Power off 2024-04-15T15:24:11.931 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T15:24:11.942 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-15T15:24:11.960 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-04-15T15:24:11.960 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:24:11.988 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-04-15T15:24:15.943 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:24:15.989 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:24:16.053 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-15T15:24:16.098 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-04-15T15:24:20.055 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:24:20.099 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:24:20.164 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-15T15:24:20.209 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-04-15T15:24:24.166 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:24:24.210 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:24:24.277 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2024-04-15T15:24:24.321 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-04-15T15:24:24.378 INFO:teuthology.orchestra.console.smithi012:Power off completed 2024-04-15T15:24:24.422 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-04-15T15:24:24.527 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} duration: 653.1029348373413 failure_reason: 'Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=23c3ed57813f4ed6818c345f16c1a286 status: fail success: false 2024-04-15T15:24:24.528 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T15:24:24.602 INFO:teuthology.run:FAIL